Srsly Risky Biz: Chipping Away at Chinese AI Risks
Your weekly dose of Seriously Risky Business news is written by Tom Uren and edited by Patrick Gray and Amberleigh Jack. This week's edition is sponsored by Airlock Digital.
You can hear a podcast discussion of this newsletter by searching for "Risky Business News" in your podcatcher or subscribing via this RSS feed.
The Trump administration has been trying to address two separate AI-related risks in recent months: The specific risk to US national security from China developing powerful AI and the global risk that powerful hacking machines will be available to all and sundry. A proposed bill suggests a sensible way for the US to chip away at both these risks, at least a little.
The Collaboration on Adversarial Threats and Security Risks Act proposes safe harbor provisions for AI companies so they can share information related to AI-specific security risks. The idea here is to encourage frontier labs to work together to counter threats from Chinese AI labs, particularly what they describe as IP theft via distillation. Without this bill, sharing this kind of information could fall afoul of anti-trust legislation that prohibits collusion.
We know the frontier labs can already detect distillation because they're always complaining about it after the fact. The idea behind this bill is that more permissive information sharing would let them respond quicker and disrupt at least some distillation campaigns.
A bill like this isn't necessarily the most effective way for the US government to prevent or deter distillation attacks. But the more direct approaches, like adding Chinese companies to the entities list, come with problems.
American companies, for example, are increasingly using open-weight AI models, the best of which are produced by the same Chinese labs that are carrying out large scale distillation campaigns.
Indeed, after Treasury Secretary Scott Bessent issued a statement saying that "Entity List designations will be on the table", a tech industry coalition of 76 companies published an open letter supporting the role of open-weight AI models in the technology mix.
So, the consensus seems to be that punitive action targeting Chinese labs could be a bonus for the American frontier AI companies, but likely a negative for the US economy writ large.
Really, though, the battle for AI dominance is being fought around hardware access, not the models themselves. The US controls chip exports, and there is some evidence that these controls are having an impact.
When AI-focussed journalists Lily Ottinger and Kai Williams toured Chinese AI labs in May this year they reported:
Basically every AI researcher we talked to in China brought up the same complaint: their companies lack sufficient access to the advanced compute resources necessary to train and run AI models.
American chip export controls are imperfect, but they're biting, and reinforcing them is the most effective way for the US to stay ahead.
In the areas around AI that aren't about staying ahead, we think there's room for the United States and China to cooperate.
When open-weight models get really good, we suspect neither government will be happy with everyone on the planet having access to powerful AI hacking machines. That time could only be a matter of months away, as the recently released open-weight Chinese Kimi K3 model is roughly equivalent to US frontier models from about six months ago, according to a joint US/UK evaluation.
China already has strict AI regulation, but it's focussed on making sure that AI does not upset the Communist Party's information control and censorship apparatus.
However, we think Chinese regulators will eventually reign in the release of very capable models. Very capable open-weight models will be used for all sorts of hacking, some of which will make the Chinese Communist Party look bad. The Party will not like that kind of political risk, so cyber safety regulation is coming.
But right now there is an opportunity for the US government to engage the Chinese government to encourage and shape this regulation to its advantage.
So as an overall strategy, what we've laid out above makes sense: Laws that free up frontier labs to combat distillation collectively; tightening chip controls to make sure the US stays ahead of China; and engaging with Chinese regulators to prevent the spread of top-tier hacking capabilities to all and sundry.
Will this be what happens? Long-term strategic thinking is not a Trump admin forte, so we'll hope for the best, but prepare for the status quo.
Iran's Perfectly Calibrated Cyber Attack
A cyber attack on Minnesota water utilities coupled with a US federal government warning raises the possibility that Iranian hackers have begun targeting US critical infrastructure in earnest.
This week more than 30 Minnesota community water systems were targeted by a "coordinated cyberattack", according to the state’s IT services. Disruptions were reported in a number of cities, but the water remained safe to drink. The worst impact reported was in the city of Braham where residents were asked to limit their consumption of water for a few hours as the city was relying on supply in a single water tower.
The attack came less than a week after US federal government agencies updated a warning about the possibility of Iranian state-affiliated cyber actors targeting US critical infrastructure. The warning cites incidents since March in which multiple victims across several critical infrastructure sectors had been compromised and that some victims "experienced operational disruption and financial loss".
There is no formal attribution of the Minnesota attacks, but security firm Tenable says the "operational pattern" is consistent with the CyberAv3ngers, a threat actor associated with Iran's Islamic Revolutionary Guard Corp (IRGC).
In March we wrote that even if the war in Iran successfully achieved the White House's stated objectives, it could still "result in an enduring increase in Iran's capacity and appetite for cyber mayhem". Cyber operations targeting the US and Israel could provide propaganda victories with very little chance of a kinetic response.
The Minnesota attacks caused minimal impact, so it would be tempting to argue that they were a flop. Given that the war is still ongoing and also very unpopular, however, we think these attacks might be perfectly calibrated. They're significant enough to generate headlines and public unease, and, maybe, make Americans wonder about the point of the entire war. But they're not lethal or damaging enough to ensure America bands together against a single enemy.
Our crystal ball says we can expect more of this.
Watch James Wilson and Tom Uren discuss this edition of the newsletter:
Three Reasons to Be Cheerful This Week:
- US visa restrictions for cybercriminals: Last week, the US Secretary of State Marco Rubio announced visa restrictions for cybercriminals including sextortionists and those running cyber-enabled scams. The policy applies to the " individuals responsible" for the crimes and, in some cases, their family members.
- AI security tools are a thing: This week Microsoft announced a security-focussed AI model that, when coupled with its MDASH harness, is cheaper and better at vulnerability identification and remediation than current frontier models. It's not this specific announcement that makes us cheerful, but rather what it indicates. Microsoft announcing a "world-class security" machine as a product means that there will likely be a number of competitors offering a variety of similar products.
- AI works on cryptographic algorithms too: Anthropic has posted about Claude Mythos Preview being used to find flaws in two different cryptographic systems. One of the results came in a proposed post-quantum algorithm that was being evaluated by NIST. This is a good example of AI being used preventatively to improve standards before they decided.
Sponsor Section
In this Risky Business sponsor interview, James Wilson chats with Airlock Digital co-founders David Cottingham and Daniel Schell about how attackers are using LLMs to enumerate EDR detections.
Shorts
North Korean Hackers Hack North Korea
The Daily NK reported last week that a group of former North Korean military intelligence operatives had hacked two of the country's banks to steal funds for their own personal enrichment.
A source told the Daily NK that the group had stolen funds from two North Korean banks, the Chosun Central Bank and the Foreign Trade Bank. The stolen funds were then converted into cryptocurrency and laundered before being converted back into cash in China. The ringleaders were cyber operations veterans formerly from North Korea's Reconnaissance General Bureau.
It looks like the outcome will be grim for the hackers. Per The Daily NK:
Officials in Pyongyang expect harsh punishment for those involved. "They used the skills the state trained them with to defend the country, and instead robbed the country’s coffers," one official said, according to the source. "This goes beyond ordinary guilt-by-association penalties. It will be hard for the entire family line to survive."
Crikey.
Risky Biz Talks
You can find the audio edition of this newsletter and other fine podcasts and interviews in the Risky Biz News feed (RSS, iTunes or Spotify).
In our last "Between Two Nerds" discussion Tom Uren and The Grugq discuss how important people are to cyber power and whether the rise of AI is changing that.
Or watch it on YouTube!
From Risky Bulletin:
New Chinese cyber contractor identified: The cyber sleuths at Intrusion Truth have uncovered a new secretive Chinese IT company that appears to work as a cyber contractor and tool developer for Chinese state-sponsored hacking operations.
Online clues appear to suggest that Guangdong Chanming appears to have developed RedRelay (aka ORBWEAVER), an ORB network (aka proxy botnet) that was used by almost a dozen Chinese APT groups to hide the origin of their attacks.
According to Intrusion Truth, the company's customers allegedly include the Chinese People's Liberation Army and the Ministry of Public Security, which manages China's police forces.
[more on Risky Bulletin]
A JSON RCE bug is about to rock the Java world: Threat actors are exploiting a vulnerability in Alibaba's Fastjson, one of the Java ecosystem's most popular libraries for working with JSON-formatted data.
Active exploitation began last week, a day after details about the security flaw were revealed by cybersecurity firm FearsOff.
The attacks, first spotted and documented by Imperva and ThreatBook, target CVE-2026-16723, a vulnerability that can enable unauthenticated remote code execution attacks against Java projects that use the Fastjson library as a component.
[more on Risky Bulletin]
Western cyber agencies warn of Russian hacks of Zimbra servers: Cybersecurity and intelligence agencies from multiple Western countries have issued joint security advisories on Thursday warning of a major Russian hacking campaign that's targeting Zimbra email servers.
The attacks have been going on since last year. The zero-day, tracked as CVE-2025-66376, was patched in November but attacks have been traced back to at least July.
The zero-day itself is a stored cross-site scripting (XSS) bug that allows the attackers to load malicious code inside a Zimbra webmail client via the CSS @import feature. The malicious code would load a web tool called Ulej (Russian for Beehive) that could be used to harvest credentials, session tokens, backup 2FA codes, browser-saved passwords, and the contents of the victim’s mailbox going back 90 days.
[more on Risky Bulletin]
