Srsly Risky Biz: Knives Are Out For Open-Weight AI Models

Your weekly dose of Seriously Risky Business news is written by Tom Uren and edited by Patrick Gray and Amberleigh Jack. This week's edition is sponsored by Thinkst.

You can hear a podcast discussion of this newsletter by searching for "Risky Business News" in your podcatcher or subscribing via this RSS feed.

Both the American and Chinese governments have signalled they plan to rein in open-weight AI models. 

The Trump administration seems certain to add some Chinese technology companies to the Entity List to protect investment in American frontier AI models. Meanwhile, Beijing is apparently weighing applying export restrictions on Chinese AI tech, including open-weight models.

Overnight, different US government officials issued a strong, coordinated signal that they plan to take action against Chinese AI companies. Michael Kratsios, the director of the White House's Office of Science and Technology Policy, wrote on X that Chinese company Moonshot AI had "developed a sophisticated internal platform to conduct large scale distillation against US models" and that "large-scale, covert industrial distillation aimed at stealing proprietary U.S. technology and undermining American research is unacceptable."

Treasury Secretary Scott Bessent also weighed in, saying on X that "sanctions and Entity List designations will be on the table". 

This has all come hot on the heels of two recent and interesting model launches out of China: GLM 5.2 from Z.ai and Moonshot AI's Kimi K3. 

Moonshot announced its new Kimi model late last week and although it isn't open-weight yet, Moonshot says it will release the weights for Kimi K3 on 27 July. 

According to independent model evaluations, it's very good. The Artificial Analysis Intelligence Index ranks the model at number three, the Vals AI index ranks it number two, and it ranks number one according to the Frontend Code Arena. 

Moonshot AI itself is more measured, saying it "still trails the most powerful proprietary models". 

No matter exactly how good Kimi K3 is, from the perspective of America's leading AI companies, having highly capable, open weight alternatives undermining your business model is A Bad Time. One source that Axios described as "close to the Trump administration" says leading AI labs or their allies lobby the admin to ban open-weights models every 3-5 months.

They're furious about their models being distilled, but we're not convinced that distillation is the whole ballgame here. 

Dean Ball, head of strategic future at OpenAI wrote on X that he didn't think the performance of Kimi K3 "can be explained away by distillation". Similarly, AI researcher and author of the Interconnects substack Nathan Lambert, also thinks the impact of distillation is overstated and that AI training pipelines are evolving so that distillation is becoming less important anyway. 

Despite that, it seems that  internal debates within the Trump administration regarding open-weight Chinese models have been settled: The Chinese are stealing American IP, and there needs to be a reckoning. 

That's a shame, because these open-weight models are currently filling a gap not served by America's frontier labs.

Take this month's hack of US AI development company Hugging Face by rogue OpenAI models. This bonkers hack was discussed extensively on this week's episode of the Risky Business podcast, but the short description is that some OpenAI models hacked the company's own infrastructure to escape a sandbox, then hacked Hugging Face, all in order to cheat on a cyber security evaluation test.

When Hugging Face tried to analyse the intrusion with frontier models it was stymied by safety guardrails. The analysis required sending real attack commands, exploit payloads and command and control artefacts to the frontier labs, which couldn't tell whether they might be helping an attacker instead of legitimate incident response.

So instead, Hugging Face turned to the Chinese open-weights GLM 5.2 model from Z.ai, running it on its own hardware. The company said:

To understand what a swarm of tens of thousands of automated actions did, we ran LLM-driven analysis agents over the full attacker action log, comprised of more than 17,000 recorded events. This allowed us to reconstruct the timeline, extract indicators of compromise, map the credentials touched, and separate genuine impact from decoy activity. Thanks to this approach, we were able to do in hours what would usually take days, and match the adversary's speed.

Using a self-hosted open-weight model also meant the company could guarantee that none of its sensitive data needed to be shipped off elsewhere. So American companies have three good reasons to turn to Chinese open-weight models: They're cheaper, you can run them on your own hardware, and they've proven capable at cyber security tasks.

That's why we don't think it makes sense for the US government to outright ban foreign open-weights models. 

The Chinese government has a say here too and there are emerging reports that Beijing is considering controlling the export of new open-weight models. 

Until now, releasing open-weights models has made sense to Chinese companies and the Chinese government. Z.ai's Director of Product Zixuan Li told the ChinaTalk substack in November last year that companies do it to contribute to research in the field, but also to build acceptance overseas. 

"I think it is necessary to be open right now for people to use our models". 

From the Chinese government's perspective, it has promoted the position that AI is a global public good that should be developed for the benefit of all humankind. But this strategy is colliding with the reality that as these models become more capable they become far more dangerous.

We don't think it will be too long before open-weight models are capable of the sort of hacking in the Hugging Face example we described previously. This involved a combination of OpenAI models, including its current best GPT‑5.6 Sol and an even more capable pre-release model. 

A report from the UK's AI Security Institute (AISI) released last week found that leading open-weight models were just four to seven months behind the frontier models, depending upon what you were measuring. That report examined Chinese open-weight models released in mid-April, with OpenAI's GPT-5.6 Sol and Claude Mythos 5. Kimi K3 appears to be a significant advance over those mid-April models. 

Of course in the Hugging Face incident, OpenAI had turned off safeguards to get a better read on exactly how good its models were. Research released in May this year shows it is possible to remove safeguards from open-weights models quickly and painlessly. So as new and increasingly capable open-weights models are released, someone will remove safeguards so that they can be misused. And thanks to OpenAI's sloppy testing sandbox we've seen what that'll look like.

So having its tech firms drop increasingly powerful models with easily removed safeguards onto the internet for Joe Public to use to cause chaos doesn't seem like a recipe that the Chinese government will be comfortable with for long. They're not exactly libertarians over there, after all.

The American and the Chinese governments each have their own motivations. But we suspect the current golden age of highly capable open-weight models dropping onto the internet every other week is coming to an end.

For Scattered Spider, the Chickens Are Finally Coming Home 

A string of successes in recent months suggests that law enforcement is finally getting a handle on the internet's brashest hackers: Teenagers.

Last week two members of the so-called Scattered Spider group were each sentenced in the UK to five and half years in prison for their roles in a 2024 breach of Transport for London's network. Thalha Jubair, 20 and Owen Flowers, 18, were described by prosecutors as leading members of Scattered Spider. In April this year, Tyler Robert Buchanan, 24, also pleaded guilty to different set of Scattered Spider-related charges. 

Early this month the US Department of Justice (DoJ) announced that another alleged member had been arrested. Peter Stokes, 19, a dual US-Estonian citizen, was detained by Finnish authorities in April and extradited to the US in late June. The criminal complaint alleges that Stokes was involved in a number of incidents, including the breaching and attempted ransom of a luxury jewelry retailer. 

The DoJ says that Scattered Spider has been involved in over 100 network intrusions resulting in more than USD$100 million in ransom payments. 

We first covered the group in 2023 in the wake of breaches at Caesars Entertainment and MGM Resorts International, two large US resort, entertainment and gaming companies. Scattered Spider shared some members with a previous group known as Lapsus$ that had been so outrageously successful that it was the subject of a US Cyber Safety Review Board report that year. 

Reporting in 2024 described the group not so much as a discrete gang, but as an ecosystem or community that shares techniques and teaches juveniles to be effective hackers. Given that its members teamed up for particular projects, we likened it to Hollywood as a collective, rather than a single production company. 

In mid-2025, however, several security firms said that a small number of key members were driving the activities of Scattered Spider, with different firms suggesting there were between two to four key players, who were essentially project managers. They selected targets and built a team of people from the broader group to carry out the attacks. 

For key Scattered Spider members, elite social engineering skills appeared to be a key skill. Ransomware incident response firm Coveware wrote in 2023 that "skillful social engineering of the IT support desk to subvert, reset or overcome multi-factor authentication" was a common tactic and that recordings confirmed that the same "two or three" individuals were consistently involved.  

Although it has taken some time, these key individuals are now getting arrested. And it could just be because they have grown up. 

In an interview with Zero Day, juvenile cybercrime expert Alison Nixon described why it is difficult to both deter and arrest young offenders involved with Scattered Spider's activities. Nixon says the FBI will not arrest juveniles because "there is no federal juvenile system". It's only once they turn 18 and can be charged as an adult are suspects arrested. As for deterrence, Nixon says steering teenagers with no concept of the future onto the straight and narrow is difficult.

At this point, several influential members of Scattered Spider are aging out, have been arrested and some even sentenced to jail. We are hopeful that it will quiet the group, at least for a while. 

Watch James Wilson and Tom Uren discuss this edition of the newsletter:

Three Reasons to Be Cheerful This Week:

1. Information Commissioner lets Qantas off the hook: The Office of the Australian information Commissioner released its preliminary inquiry into the June 2025 data breach at Australian airline Qantas. The OAIC doesn't plan to launch a full investigation because it thinks Qantas had taken "reasonable" steps to protect its data and had responded quickly to remediate the breach. The incident has all the hallmarks of a Scattered Spider attack, so the good news here is that it is possible to get owned by kids and yet somehow avoid the ire of the regulator. 
2. Kratos takedown: German and US authorities have taken down the Kratos phishing-as-a-service operation and disrupted 200 servers. Its developer was also arrested by authorities in Indonesia.  
3. Krebs on Security fixes LG TVs: LG has committed to ridding its webOS smart TV store of apps that turn a consumer's television into a residential proxy node. The promise came after Krebs On Security highlighted research that found 42% of webOS smart TV apps enrolled the devices into these networks. Residential proxies are often used for malicious purposes.    

Sponsor Section

In this Risky Business sponsor interview, Casey Ellis chats with Haroon Meer from Thinkst about building companies customers don’t hate. Haroon explains why Thinkst still offers Canary tokens for free and why it has avoided annual price hikes on its paid products. They talk about Eric Ries’s “Incorruptible”, Rob Lee’s 100-year-company approach at Dragos, and why keeping customers happy is a better business strategy than chasing easy sugar highs.

Risky Biz Talks

You can find the audio edition of this newsletter and other fine podcasts and interviews in the Risky Biz News feed (RSS, iTunes or Spotify).  

In our last "Between Two Nerds" discussion Tom Uren and The Grugq discuss just how important exploits are for cyber operations using data published in a new paper authored by two members of Ukraine’s cyber security agency.

Or watch it on YouTube!

From Risky Bulletin:

Linux kernel discloses 442 CVEs as AI bugpocalypse settles in: The Linux kernel project has disclosed 442 vulnerabilities over the past three days, in a massive dumb of CVEs on its security mailing list.

Although not confirmed, the bugs were likely discovered using AI tools. Over the past months, projects like Anthropic's Glasswing and OpenAI's Daybreak have been granting access to advanced frontier cybersecurity models to top-tier security firms and researchers to find bugs with AI in major open-source projects.

[more on Risky Bulletin]

Hacker wipes Romania's entire land registry database: A hacker has breached Romania's cadastre agency and wiped the country's entire land registry database following a failed extortion attempt.

The hack has brought Romania's entire real-estate market to a standstill as official apps and websites have been offline for a week. Notaries can't record new transactions while citizens can't obtain proof of ownership or detailed land records.

Email servers at the National Agency for Cadastre and Real Estate Advertising (Agenția Națională de Cadastru și Publicitate Imobiliară, or ANCPI) were also down as part of the incident.

Sources told Risky Business that the hacker entered using valid credentials, mapped internal systems, and wiped systems and backups after failing to extort the agency.

[more on Risky Bulletin]