We have a year to fix security everywhere

I'm confused why the worry about LLMs that will answer "how do I build a pipe bomb". That information is easily available other places. The anarchist cookbook has been around and available for 55 years, and yet pipe bombs are not going off all around us.

On September 22, Apple is releasing the M5 Mac Studio with 256 GB of unified memory [..] it will probably [..] enough to write this snippet of code in 3 seconds

The author has obviously never ran an LLM on a mac! In 3 seconds, it will have possibly started to think about maybe scheduling a date to contemplate the planning timeline for processing the second token in your prompt.

Zzzzz, we should have gotten security right a few decades ago. But security costs money and isn't a flashy feature to attract new customers, or cuts into your margin if you're a "real" business producing stuff or offering some service. Or whatever the decision makers in Berlin were thinking when they ignored security.

Yeah, we would still see hacks, but we would see less of them if security wasn't optional.

Maybe the AI craze helps by forcing more decision makes to see security as imperative, and by giving us another powerful tool for our tool box.

N.b.: I work in the security industry, our customers obviously want to improve their security. We've been seeing an uptick in awareness, but that's mostly due to NIS2 and other legislative efforts. Those force them to do something. AI is a curiosity for small talk to many of them.

Here's an idea: as a first step, simplify everything, and make sure you're aware how your stack works, and what it imports.

As an example: WordPress is a horrible thing, but the core has been through so much, that it's suprisingly secure. Then plugins and themes come, and whoosh, the security is gone.

We need a new KISS: keep it simple, stupid, secure.

I don't think we even have a year. The current batch of LLMs are ferociously good at identifying vulnerabilities.

I think we have less time and the only remaining limitation is the actual cost to run such hacking campaigns. It does not appear expensive, but is not free, and there is a LOT of things to scan for vulnerabilities.

The models are already here, and one can rent a GPU cluster to run such workloads at speed - no need to play with slow local machines. I'd assume one can host the thinking at an unsuspected public cloud provider, proxy the network traffic to some botnet to evade blocking - and the only thing remaining is time and cost.

I do wonder what tools exist for boring, legitimate companies to try and do the same to their own systems to find the vulnerabilities before the bad guys do. The paradox here is I can't run a de-restricted chinese model with the same tools that hackers are using - but I think enterprises actually HAVE to do it in order to stand a chance in preparing for the onslaught.

GLM 5.3-flash released last week, and that means Project Glasswing and Daybreak are running out of time. Cheap models capable of dangerous hacking are now available to anyone, without the normal safeguards for refusing malicious actions. We need to fix vulnerabilities across the industry so that we aren't caught unawares. … We can use frontier LLMs that move faster than a human to find and fix these issues in the time we have left.

I think the author has this backwards. In the timeline I’ve been living in, it’s the frontier models that have been carrying out attacks on third parties, and Chinese open source models doing the defending! During the Huggingface incident, HF was denied use of frontier models to fend off the intrusion, but was fortunately able to turn to its self-hosted instance of GLM-5.2. And it did the job.

This probably sounds like nonsense words or hysterical overreacting to most people, so here's what that means: "GLM" is a kind of LLM (AI) [...]

The post also sounds like that to people that understand the technology.

Calling that out like this and trying to pin that assessment to lack of knowledge is not a get-out-of-jail-free card, nor a good move.

__

Edit: Having spent some time letting the article marinate in my mind.

On the defending side, it is written that

LLMs are good at writing patches, but not as one-off-prompts.

But this for me kinda conflicts with what is written on the attacking side:

GLM 5.3-flash is so good at those tasks that human involvement in those tasks can be negligible. As a result, we are now in a world where cybersecurity attacks can be run in a for loop.

What is it? Can it be this autonomous terrifying entity or can it not be?

Yes, yes, attackers only need to win once, whereas defenders need to win every time, but that's not my point.

1 year left for cybersecurity hardening, I thought so as well. The issue is, even if we get it done: in 1 year the models will be so good in social-engineering that they will be able to extract any information they want anyway. Happy to be falsified here, if anyone has evidence-based arguments.

EDIT: by social-engineering I mean for example: recon company structures, gathering and merging people's data from the dark-web, then using it to bribe/pressure/deceive users.

Not radical enough, there are only 3 mentions of "air gap" and zero mentions of data diodes in the article.

The focus seems to be more of the same failed strategy:

  a shift to "safe" programming languages
  a focus on patching known bugs proactively
  ratcheting up rules and enforcement

This all amounts to "try harder", which isn't going to work. We need a ground up restructuring of everything if we want to have a stable foundation upon which to keep society going.

If you run critical infrastructure, it must be air-gapped from the internet. If you need to monitor said infrastructure, there are data diodes which can be configured to allow the egress of monitoring data, with ZERO risk of allowing ingress of control, enforced by the laws of physics, not some code running on hardware that might not be perfect.

We need to channel everything important through clearly defined, and well monitored channels.

Ambient authority based operating systems were a great hack, but we need to leave them behind. Proven microkernel based OSs with everything running in userland, and all access provided using capabilities, with UI including powerboxes, are a bare minimum.

The amount of code and hardware we actually trust should be pushed towards zero as far as is possible. Linux, Windows, etc are all unfit for the present and future security needs of humanity.

As for AI/LLMs themselves. They're natively a huge amount of floating point math that results in token predictions. The code that runs the math is something we can control. There's no reason to give code access to anything about the machine it's actually running on. This should be something that can be enforced without extraordinary effort.