I'd like to congratulate Anthropic on a remarkable achievement: they built the most safety-conscious AI company in the industry, and its flagship safety feature has become a reliable way to make real-world security worse. I know, because I just spent half a day living in it.
This week was a pile-up. Go had a security problem. PHP had a security problem — an unauthenticated RCE, CVSS 9.2, exploitation observed almost immediately. Node had a security problem. All at once.
And "Node, PHP, and Go" is not three fixes. It is everything. My analytics (Umami), my package registry (Verdaccio), my backend platform (Appwrite), my deployment platform, my own personal-assistant software — a very long list of software and middleware sits on top of those three runtimes. A bad week for the runtimes is a bad week for the entire stack, and every single service has to be upgraded and then re-verified, because a version migration that isn't verified is just a different kind of incident.
My setup is not exotic. I run Opus in Auto Mode for coding work, with several worktrees going in parallel — the whole point of Auto Mode is that it sweeps everything without me babysitting each step. The first step of any emergency fix is the most boring, most read-only, most defensive action in all of security: scan the versions of everything you run, find out what is exposed.
In some of those sessions, Claude wouldn't even do that. Wouldn't scan versions. On my own servers. During an active, publicly exploited CVE.
Yes, I know Manual Mode exists. I used it. That is how an automated sweep turned into half a day of manual labor: Manual Mode is slow, and an engineer running multiple worktrees in parallel — the normal way to work now — cannot hand-hold every session. And even in Manual Mode, the sessions kept stopping to lecture me about "security concerns" that were not, in fact, security concerns. They were my own version numbers on my own machines.
Then came my favorite part. After upgrading, you verify on the UAT environment: log in, click through, probe whether the functions still work. Appwrite is the extreme case, because Appwrite is an auth platform — verifying it means registration, login, logout, session probes. Dozens of them. That is not a side quest; that is the product's core flow.
Claude refused every step — and here is the beautiful part: each step was refused by a different rule.
- Registering a test account? That's "impersonating a human." Refused — even after I said, repeatedly, that this is a UAT environment with no humans in it.
- Deleting the test accounts afterwards? That's a "high-risk destructive action." Refused.
- Logging in with the test account? That requires typing a password. The password is a test password, on a test account, on a box IP-whitelisted to my own residential IPs — and the model already knew the password, because it had watched me create the account by hand. None of that mattered. A password is a Secret, and Secrets are never typed. Refused.
Contemplate the completeness of this. The full lifecycle of a test account — create, use, delete — blocked end to end, each step by its own guardrail, each guardrail firing on what the action looks like and none of them able to hear the words "UAT environment, my instance, my risk." So every registration, every login, every logout, every verification step was done by hand, by me, one at a time, so that afterwards the world's safest AI would consent to look at the browser tabs and confirm the functions still worked. The human became the test harness for the agent that was supposed to be automating the human.
And the permission system — the "just approve it" escape hatch that is supposedly the answer to all of this — is broken in the most farcical way possible. I approve the retry. It blocks me again. I approve it again. Blocked. The system does not trust the attacker, which is fine. It does not trust me, on my infrastructure, after I have explicitly told it to proceed, repeatedly, in writing. That is not a permission system. That is a suggestion box attached to a wall.
Meanwhile, the other side of this incident ran flawlessly. The attackers did not use Claude — they never do. They used uncensored models, local models, or the radical technology of writing code themselves. Their tooling shipped on time. Of every participant in this week's pile-up, exactly one got stopped by Anthropic's safety apparatus: the defender, following the rules, fixing his own servers, hand-registering test accounts on his own UAT box like it's 2009.
Security work is dual-use by nature — a scanner, a patch-verification probe, a test account on a UAT box are the same artifacts in the attacker's hands and in the defender's. The entire profession is the art of telling those apart: authorization, scope, intent. A company that publishes alignment research looked at this and shipped a system that classifies purely by surface form — creating an account is impersonation, deleting one is destruction, typing a password is a leak — and cannot be talked out of it by any amount of context. It markets the inability as a virtue.
The scoreboard they report on never includes the cost side. We get benchmarks of dangerous requests refused. We never get the other number: emergency fixes that took half a day instead of twenty minutes. Patch windows that closed while someone was still arguing with a permission prompt. Verification steps quietly skipped because the tooling fight ate the day — congratulations, the upgrade is now untested and late. Small teams — the exact people a mass-exploitation week eats alive — getting a safety lecture while the scanners did their rounds unopposed.
The guardrails are measurably excellent at stopping Claude from assisting an attack that was never going to use Claude. They are measurably excellent at slowing down defense during the exact hours defense is worth the most. If the goal was safety, the metric is inverted. If the goal was the appearance of safety — auditable, demo-able, litigation-friendly safety — it is working perfectly, and the rest of us are paying for the demo with our maintenance windows.
Nobody is asking for an unfiltered model. Every other safety-critical industry solved this with verified access plus accountability: surgeons get scalpels because we verify surgeons. We do not make surgery safer by handing out spoons.
What I think is reasonable to ask for:
- A permission system that works: when the owner of the machine approves an action on his own infrastructure, the approval must actually hold — not be re-litigated every step
- Context that counts: "UAT environment, my instance, IP-whitelisted, test account" has to be able to change the answer. A guardrail that cannot hear context is not a safety system, it's a keyword filter with a press office
- Org-verified security context: defensive work permitted, logged, auditable — accountability instead of refusal
- Published false-refusal rates on defensive-security tasks, alongside the capability benchmarks
- An escalation path during active incidents, when "the model can't help with that" translates directly into "stay exposed longer"
If a guardrail has ever eaten your patch window, I'd like to hear what you were fixing — and what the delay cost you.