Risky Biz News: CrowdStrike faulty update affects 8.5 million Windows systems
In other news: US sanctions two Russian hacktivists; MGM hacking suspect detained in the UK; two LockBit members plead guilty.
This newsletter is brought to you by Socket, a developer-first security platform that prevents vulnerable and malicious open-source dependencies from infiltrating software supply chains. You can subscribe to an audio version of this newsletter as a podcast by searching for "Risky Business News" in your podcatcher or subscribing via this RSS feed. On Apple Podcasts:
Around 8.5 million Windows systems went down on Friday in one of the worst IT outages in history.
The incident was caused by a faulty configuration update to the CrowdStrike Falcon security software that caused Windows computers to crash with a Blue Screen of Death (BSOD).
Since CrowdStrike Falcon is an enterprise-centric EDR, the incident caused crucial IT systems to go down in all the places you don't usually want them to go out.
Outages were reported in places like airports, hospitals, banks, energy grids, news organizations, and loads of official government agencies.
Planes were grounded across several countries, 911 emergency systems went down, hospitals canceled medical procedures, ATMs went offline, stock trading stopped, buses and trains were delayed, ships got stuck in ports, border and customs checks stopped, Windows-based online services went down (eg. ICANN), and there's even an unconfirmed report that one nuclear facility was affected.
The Mercedes F1 team, where CrowdStrike is a main sponsor, had to deal with the aftermath, hindering engineers from preparing the cars for the upcoming Hungarian GP. Heck, even Russia had to deal with some outages.
It was a clusterf**k on so many levels that it is hard to put into words how much of the world was upended on Friday, with some outages extending into the weekend.
Reddit is full of horrible stories where admins lost their jobs, faced legal threats, or were forced to sleep at their workplace to help restore networks. There are reports of companies having tens of thousands of systems affected by the update.
The recovery steps aren't a walk in the park either. It's not like CrowdStrike or Microsoft could have shipped a new update and fixed things in the span of a few minutes.
Instead, users had to boot Windows into Safe Mode and search and delete a very specific file. The recovery cannot be fully or remotely automated and an operator must go through the process on each affected system.
Microsoft has also released a recovery tool which creates a bootable USB drive that IT admins can use to quickly recover impacted machines—but an operator still needs to be in front of an affected device.
For some super lucky users, the BSOD error corrected itself just by constantly rebooting affected systems. Apparently, some systems are able to gain short access to networking capabilities to download the fixed CrowdStrike update file and overwrite the old buggy one. However, this is not the universal recommended fix. There are people reporting that they managed to fix their systems after three reboots, while others needed tens of reboots.
It took hours for the debug information to make its way downstream, meaning some of the world's largest companies had to bring their businesses to a halt, losing probably billions in the process. (extremely rough estimation, but probably in the correct range)
Unfortunately, the internet is also full of idiots willing to share their dumb opinions. In the year of the Lord 2024, we had people argue that it's time to ditch security products since they can cause this type of outage. Yes, that's the solution. [eyeroll]
There's also this garbage trending on Twitter—because that's what Twitter is these days.
But going back to the topic, CrowdStrike's blunder is not unique or new, for that matter. Something similar impacted loads of other vendors before, from Panda Security to Kaspersky and McAfee. Ironically, CrowdStrike founder and CEO George Kurtz was McAfee's CTO at the time, but don't go spinning conspiracy theories about it. It doesn't actually mean that much.
Stuff like this tends to happen, and quite a lot. As an infosec reporter, I stopped covering these antivirus update blunders after my first or second year because there were so many, and the articles were just repetitive.
Most impact only a small subset of users, typically on a particular platform or hardware specification. They usually have the same devastating impact, causing BSOD errors and crashing systems because of the nature of security software itself, which needs to run inside the operating system kernel so it can tap into everything that happens on a PC.
CrowdStrike released an initial post-mortem report of the faulty update on Saturday. It blamed the issue on what the company calls a "channel file" update, which are special files that update the Falcon endpoint detection and response (EDR) client with new techniques abused by threat actors.
In this case, it was "Channel File 291" (C-00000291*.sys) that caused the crashes.
CrowdStrike says this file was supposed to update the Falcon EDR to detect malware that abuses Windows named pipes to communicate with its command and control (C2) server.
Such techniques were recently added to several C2 frameworks—tools used by threat actors and penetration testing teams—and CrowdStrike wanted to be on top of the new technique.
The company says the Falcon update file, unfortunately, triggered a logic error. Since Falcon ran in the Windows kernel, the error brought down the house and caused Windows to crash with a BSOD. After that, it was just a house of cards. As the update was delivered to more and more CrowdStrike customers, the dominos started falling all over the world.
Kurtz was adamant on Friday that this was just an error on the company's part and made it explicitly clear that there was no cyberattack against its systems. US government officials also echoed the same thing.
For now, CrowdStrike seems to be focused on bringing its customers back online. The incident is likely to have some major repercussions going beyond the actual technical details and the global outages. What they will be, I cannot be sure, but I smell some politicians waiting to pounce on it with some "ideas."
This might also be the perfect opportunity/excuse for Microsoft to go with Apple's route and kick most security vendors and drivers out of the kernel.
But before that, Microsoft might need to convince the EU to dismiss a 2009 agreement first. Per this agreement, Microsoft cannot wall off its OS from security tools.
The EU and Microsoft reached this arrangement following an anti-competitive complaint filed by security software vendors after Microsoft entered the cybersecurity and AV market with Defender, with vendors fearing Microsoft would use its control over Windows to put everyone out of business by neutering their products.
After the recent Chinese and Russian hacks of Microsoft cloud infrastructure, we now know very well what happens when Microsoft has a dominant market position, and it's never a good thing, so the existence of this agreement isn't such a bad idea.
If Microsoft wants to kick security software out of the kernel, Defender needs to lose it too. Unfortunately, blinding security tools to the kernel now puts everyone in the iOS quandary, where everyone loses visibility in what happens on a system. That's not such a good idea, so we're back with this argument where we started.
In closing, just be aware that threat actors are registering hundreds of CrowdStrike-related domains that will most likely be used in spear-phishing and malware delivery campaigns. It's honestly one of the best and easiest phishing opportunities they've had in a while.
Breaches, hacks, and security incidents
Rho Markets crypto-heist: A threat actor has stolen $7.6 million worth of assets from DeFi platform Rho Markets. Security firm Cyvers says the attacker gained access to one of Rho Markets' blockchain oracles. An oracle is an endpoint that allows third parties to interact with the blockchain. They are typically used to support the execution of smart contracts. Since 2011, cryptocurrency companies have lost almost $19 billion across almost 800 hacks.
a16z leak: The website of American venture capital firm Andreessen Horowitz leaked details on its portfolio of companies. The company fixed the bug at the end of June and refused to pay a bounty to the researcher who found the issue. The company has $42 billion in assets under management across multiple funds. [Additional coverage in TechCrunch]
Hellenic Cadastre attacks: Greece's national cadastre agency says it was the victim of a cyberattack. The incident took place on Friday and hackers managed to successfully steal 1.2 GB of data. Hellenic Cadastre says the stolen files did not contain any information on Greek citizens.
FleepBot hack: Telegram automation service FleepBot says it was hacked over the weekend and abused to post messages to customer accounts. The incident primarily impacted Ukrainian Telegram channels. The attackers posted messages urging Ukrainians to lay down their arms and surrender. FleepBot says it disabled its bot following the attack. The FleepBot service is managed by a Russian company based in the city of Volgograd. [Additional coverage in UNN]
General tech and privacy
CoSAI: Some of the world's largest tech companies have announced the creation of a new initiative to improve the security of AI systems. The new initiative is named CoSAI, or the Coalition for Secure AI. Its main objective will be to create and release open-source standards and tools for secure-by-design AI systems. Founding members include the likes of Google, NVIDIA, Intel, Microsoft, and IBM.
Safari Private Browsing 2.0: Apple has announced an overhaul of the Safari Private Browsing mode. The new feature will include link tracking and fingerprinting protection, and the blocking of CNAME-cloaked trackers. It will also cap the lifetime of cookies and partition SessionStorage and blob URLs for each browser tab. These latter features will also be available for the main Safari browser mode.
Chunghwa Telecom incident: Taiwanese certificate authority Chunghwa Telecom has gotten in some trouble with Mozilla for breaking a bunch of CA/B Forum rules. A possible ban is incoming and the Taiwanese company is trying to argue that some of its certificates are being used in the country's critical infrastructure and may lead to outages at railways, airports, and a bunch of sensitive places.
Nigeria fines Meta: The Nigerian government has fined Meta $220 million for breaking the country's antitrust and data privacy laws. The fine comes after a 38-month investigation by Nigeria's Federal Competition and Consumer Protection Commission. The agency began investigating Meta in May 2021. It said Meta abused its market dominance position, shared Nigerians' personal data without their authorization, and denied Nigerians the right to determine how their data is used. [Read the fine document/PDF]
ICANN warns TOP TLD: ICANN has given the company behind the .top top-level domain (TLD) name until August 15 to remedy a series of technical and security issues. ICANN says .TOP Registry has failed to implement a mechanism to handle abuse complaints and domain name disputes and is also behind on payments. If .TOP Registry fails to act, it risks losing control of the entire TLD. The .top domain is the fourth-largest TLD by registrations, with approximately 3.2 million domain names. [Read ICANN's notice of breach/PDF] [Additional coverage in DomainNewsWire]
ASF to change its logo: The Apache Software Foundation says it will retire its "feather" logo in response to a complaint made last year by Natives in Tech, a US 501(c)(3) charitable organization supporting Native American tech workers. A new logo is expected in October this year. The ASF will not change its name.
Google to retire goo.gl: Google is retiring its goo.gl URL shortening service. All goo.gl links will start showing an interstitial warning of the EoL on August 23 this year. All Google shortened links will stop working after August 25, 2025.
Government, politics, and policy
DMCA exemption: The US Department of Justice has proposed a new exemption to the Digital Millennium Copyright Act (DMCA) that would allow security researchers to break the ToS of AI companies and analyze AI systems for security flaws and other good-faith research. [Additional coverage in 404 Media]
Sponsor section
In this Risky Business News sponsored interview, Tom Uren talks to Feross Aboukhadijeh, CEO and Founder of Socket about how open source repositories are riddled with horrible software. Feross explains why it makes a difference if a package is vulnerable, malicious or just unwanted and how current transparency mechanisms such as CVEs and the NVD just aren’t suitable for the challenge of open source repositories.
Cybercrime and threat intel
Two LockBit affiliates plead guilty: Two Russian nationals pleaded guilty last week in a US court to launching attacks and extorting victims with the LockBit ransomware. Mikhail Vasiliev, 34, operated as a LockBit affiliate under the names of BETTERPAY, OffTitan, and EastFarmer. Officials say he earned at least $1.9 million in ransom payments from at least 12 victims he extorted over the past five years. Ruslan Magomedovich Astamirov, 21, operated as a LockBit affiliate using the names Ghostrider, Free, and DigitalOcean. Vasiliev and Astamirov are two of the six LockBit members charged in the US for their crimes. They were detained in November 2022 and June 2023, respectively. The rest of the LockBit suspects are still at large. US authorities disrupted the LockBit gang's operation in February this year. The disruption led to a slowdown in attacks after the LockBit gang was the most active group throughout 2022 and 2023.
MGM hacking suspect detained in the UK: British police have arrested a 17-year-old boy from the city of Walsall on hacking and blackmail charges. Officials say the teen was part of a large hacking community that targeted major companies across the globe. The teen is suspected of being a member of the Scattered Spider group and of having participated in the MGM Resort hack and ransomware attack last year.
DDoS booter admin sentenced to 21 months: A university student from Scotland has been sentenced to 21 months in jail for developing and running a DDoS-for-hire service. Officials say 21-year-old Amar Tagore was behind a DDoS service named Myra. The site had hundreds of customers, and Tagore allegedly made nearly £45,000 from the service over the course of almost three years (between January 2020 and November 2022). Tagore was detained after the Myra service was used to launch DDoS attacks against the website of a job center in the city of Braintree, England, for almost three months. [Additional coverage in the Shropshire Star]
NoName057(16) arrested in Spain: Spanish authorities have arrested three individuals for allegedly helping the pro-Kremlin NoName057(16) hacktivist group launch DDoS attacks. The suspects were detained last week in the Spanish cities of Seville, Huelva, and Manacor. The three men most likely installed the group's DDoSia tool on their computers. The NoName057 then used their systems to launch DDoS attacks against Spanish government sites and other NATO countries. [h/t DataBreaches.net]
CARR sanctions: The US Treasury Department has sanctioned two Russian nationals and members of the Cyber Army of Russia Reborn hacktivism group. Sanctions were levied against the group's leader, Yuliya Vladimirovna Pankratova, and its primary hacker, Denis Olegovich Degtyarenko. Officials say the sanctions are for attacks the group launched against SCADA systems at US critical infrastructure. In January this year, the group claimed responsibility for hacking the control panels of water utilities in Abernathy and Muleshoe, Texas. The attack caused water storage tanks to overflow. The group also attacked water supply, hydroelectric, wastewater, and energy facilities in the US and Europe.
New Revolver Rabbit group: A new threat actor named Revolver Rabbit is behind a new type of operation that provides malicious domains on command for other cybercrime groups. The group has registered and is currently managing more than 500,000 domain names. Security firm Infoblox says the domains are being used by the operators of Windows and macOS infostealers.
dirDevil technique: TrustedSec has published details on a technique called dirDevil for hiding malicious code and data within folder structures.
Swap file abuse: Now, I've seen it all. Web skimming scripts hidden in server swap files.
New npm malware: Seventeen malicious npm packages were discovered last week. Check out GitHub's security advisory portal for additional details.
Threat/trend reports: CyFirma, F-Secure, Fortinet, ITRC, Radware, VulnCheck, and ZeroFox have recently published reports covering infosec industry threats and trends.
Malware technical reports
Play ransomware: Trend Micro has published a technical analysis of the Play ransomware gang's new Linux ESXi encrypter. Researchers also say the Play gang is now also using server infrastructure linked to the Prolific Puma threat actor.
Sponsor Section
Come meet the Socket team at BlackHat and DEF CON! We're sponsoring some fun networking events, and we would love to see you there.
APTs, cyber-espionage, and info-ops
Chinese APT side-hustle: Mandiant's Dan Kelly has published a Twitter post about how one member of a Chinese APT hacked dozens of MMORPG gaming companies. Kelly says the individual appears to have been running a secret game cheating service that used his access to the gaming company's database to increase in-game currency for users—some of which were Twitch and YouTube streamers.
TikTok info-ops takedowns: TikTok disrupted two influence operations active on its service last month, one from Sweden and one from Ecuador.
Vulnerabilities, security research, and bug bounty
1Panel zero-day: The 1Panel project has fixed a zero-day vulnerability (CVE-2024-39911) in its web hosting management software. The patch comes more than six weeks after security researcher Juneha published details about the vulnerability online. The bug allows threat actors to use an SQL injection attack and take over Linux servers managed through a 1Panel web hosting panel. The 1Panel software comes with open-source and commercial versions and is popular in the Chinese market.
Infosec industry
SummerCon 2024 streams: Recorded live streams from the SummerCon 2024 security conference, which took place over the weekend, are available on YouTube [Day 1, Day 2].
SteelCon 2024 streams: Recorded live streams from the SteelCon 2024 security conference, which took place over the weekend, are available on YouTube [Track 1, Track 2, Track 3].
Risky Business Podcasts
In this edition of Between Two Nerds, Tom Uren and The Grugq discuss Shashank Joshi's notes from a recent Oxford Cyber forum. Topics include the role of zero-days and who is ahead when it comes to offensive cyber operations.
In this podcast, Tom Uren and Adam Boileau talk about how countries are using cyber security reports and advisories to win friends and influence people; why having gaping holes in US federal government security is situation normal; and efforts to make up for the disappearance of Twitter's trust and safety team.
This Soap Box edition of the show is with Mike Wiacek, the CEO and Founder of Stairwell. Stairwell is a platform that creates something similar to an NDR, but for file analysis instead of network traffic. The idea is you get a copy of every unique file in your environment to the Stairwell platform, via a file forwarding agent. You get an inventory that lists where these files exist in your environment, at what times, and from there, you can start doing analysis.
