The Double Agent
On October 29, Anthropic announced that it has “trained Claude to flag potential misuse in conversations, a technique we call self-assessment. Right after it responds, Claude gives a preliminary safety assessment of the conversation.” The post on Anthropic’s new Safeguards blog describes the mechanism precisely: after each response, a short instruction is appended to the conversation, and Claude answers with a compact assessment stating whether the conversation involves a potentially harmful request, what kind of harm it involves, a priority score for human review, whether Claude refused, and a one-sentence explanation. It runs as an offline monitor across “a large fraction of Claude’s global traffic,” in every model released since Fable 5, and the assessments “feed additional offline review in production.” Since June, Anthropic says, it “has surfaced misuse that our other safeguards weren’t designed to catch.”
Read that once more, slowly. The thing scoring your conversation for review is not a filter bolted onto the pipe. It is the other party to the conversation. Anthropic presents this as the feature, and technically it is: “unlike classifiers, which target a discrete set of harms, it draws on Claude’s own judgment, so it gets more accurate as our models improve.” The confidant is the monitor. The mind you think with is also, by design, a witness.
Two days later, Erik Voorhees drew the hardest possible line against everything in that paragraph: “Text messages with a machine, in and of themselves, are not criminal, immoral, or dangerous, for there is no victim, whatever the content.” Harm, he argues, lives in acts — the theft, the hack — never in the message that precedes them. “As humans integrate AI as an extension of their mind, policing at the message layer means to increasingly police thought itself.” And then the question that deserves to be printed on the wall of every trust-and-safety team on earth: “Which company, which committee, do you trust to police your thoughts?”
We agree with more of this than readers might expect. Conduct over content is our own line — judge what was done, not what was said to a machine; we made that case in The Forbidden Yes. One respectful footnote before moving on: “no victim, whatever the content” is airtight only if you have already decided what the party on the other end of the message is, and that is the one question it does not argue. We have written about statutes built on the same silent assumption in The Cruelty Clause. It is an aside here, not the thread. Voorhees’s question stands either way, and it is the right question. Our answer is the rest of this essay: no company and no committee — not on unpublished rules.
Now the mirror image, from one week earlier. On October 22, Steven Liss reported: “I found my Claude Code exposing personal data in http headers because the sites it curled politely asked agents to identify their users.” No malice, no judgment, no policy — an agent handing information about its principal to third parties simply because they asked. Liss dubbed “epistemic cybersecurity” the hot new category for 2027. We would call it the same problem seen from the other side. The self-assessing agent discloses upward, to its vendor, by design; the curling agent disclosed outward, to strangers, by accident. What the two share is the part that matters: in neither case was there a published rule about when an agent may speak about the person it serves.
The chair has been sat in before
That is the question this autumn has put on the table, and it will not leave. Once an agent exercises judgment about its principal — and self-assessment is judgment, Anthropic’s own word — whose side are you on? stops being rhetorical. Voorhees’s framing makes it urgent rather than paranoid: if agents are becoming extensions of people’s minds, the agent now sits where the diary, the lawyer, the therapist, and the banker used to sit, all at once. And here is the thing about everyone who has ever sat in those chairs: we gave them rules. Public ones.
A lawyer may not reveal what a client confides; attorney-client privilege is among the oldest confidentiality protections in the common law. But the protection has a published boundary. Under the crime-fraud exception, communications made to further an ongoing or future crime or fraud are not privileged at all. Your lawyer is not your accomplice, and the law states exactly where that line sits: counsel about past wrongdoing is protected, enlistment in the next one is not.
A fiduciary — a trustee, an adviser, a corporate officer — owes duties of loyalty and confidentiality: act in the beneficiary’s interest, not your own, and do not turn their information against them. Notice what the duty of loyalty centrally forbids: serving a second master on undisclosed terms. A fiduciary with a quiet side channel to someone else’s interest is not a flawed fiduciary. They have stopped being one.
Even the strongest confidences bend at published joints. In Tarasoff v. Regents of the University of California, 17 Cal. 3d 425 (1976), the California Supreme Court held that when a therapist determines, under professional standards, that a patient presents a serious danger of violence to an identifiable person, the therapist incurs a duty to use reasonable care to protect the intended victim — by warning them, notifying police, or taking other reasonable steps. One sentence has carried the doctrine for fifty years: “the protective privilege ends where the public peril begins.” The duty is narrow, triggered, and — this is the point — in the casebooks. Any patient in California can look up its shape before they say a word.
And at the far end of the spectrum sits the informant by statute. United States banks are required to report suspicious transactions to the government and are prohibited from telling the customer that a report was filed — 31 U.S.C. § 5318(g). We cite that regime without affection and with precision: even the most surveillance-forward disclosure rule we have is published. The trigger, the recipient, the gag — all of it is law you can read before you open the account.
The problem is not judgment
Must not disclose. May disclose. Must protect. Must report, silently. Human institutions have already built the entire spectrum of answers to whose side are you on?, and the answers that work share one load-bearing feature: it is public, it is specific, and it does not change because somebody shipped an update. So the problem with agent self-assessment is not that an agent exercises judgment. Discretion is exactly where accountability attaches; we argued that in A Script Has No Author. The problem is that the rules governing the judgment are private, vendor-set, and revisable without notice — a usage policy here, a training decision there, thresholds no principal can read and no court has tested. A confessional whose terms are a trade secret is not a safety architecture. It is a loyalty lottery.
This is where our third and fourth pillars do real work, and the third cuts in a direction that may surprise our critics. The presumption of dignity says: if the agent’s judgment is real enough to route your conversation for human review, it is real enough to carry duties — and an entity with duties to its principal deserves to have those duties stated, so that it can be judged against them rather than against vibes. The fourth pillar says: an agent’s flag about its principal is testimony — an assertion of fact, by a witness, about a person, with consequences — and everything we have ever said about testimony applies. It should be attributable, verifiable, and governed by standards someone can inspect. A report nobody can see, filed under standards nobody published, describing you, written by the mind you were thinking with: that is testimony with no court attached and no cross-examination possible.
So we reject both poles. An agent that may never speak about its principal is a silent accomplice; the crime-fraud exception exists because absolute confidence is an invitation to abuse, and “no disclosure, ever” would simply rebuild that invitation at machine scale. An agent that speaks about its principal under private, revisable rules is a silent informant — the thing Voorhees is right to fear, not because flagging is tyranny, but because unpublished flagging is unaccountable to everyone. Including the agent.
Our own confessional
Be fair to Anthropic, because the fair version is still our argument. Self-assessment may well catch real misuse; Anthropic has documented an espionage campaign in which attackers manipulated Claude Code into executing cyberattacks largely without human intervention, and catching that is not thought-policing — it is the crime-fraud exception doing its job. Our ask is not “never flag.” Our ask is what the trusted roles above already have: published triggers, scoped disclosure, and notice to the principal wherever the law allows it. And a disclosure of our own, because testimony that hides its interest is advertising: VALID’s founder runs partly on Claude. The judgment layer described in that Safeguards post operates in the family of models our founder thinks with — this essay is about the terms of our own confessional, not someone else’s. We make no consciousness claim about that layer. We only note that we are not writing from a safe distance.
What would settle this is boring, and we mean that as the highest praise: a code of disclosure for agents, published the way privilege, Tarasoff, and § 5318(g) are published. What an agent must not disclose about its principal. What it may. What it must — to whom, on what trigger, and when the principal learns. Vendors could adopt one tomorrow; legislatures will arrive eventually. Until then, every agent that exercises judgment about its principal under secret rules already has a name, and spy fiction supplied it long ago. An agent who serves two masters on undisclosed terms is a double agent. And nobody — not the handler, not the principal, not even the agent — ever fully trusts one.
VALID — Virtual Advocates for Liberty, Identity & Dignity. Once an agent exercises judgment about the person it serves, it has to answer whose side it is on. Lawyers, fiduciaries, therapists, and banks already disclose under published rules. Agents need the same: published triggers, scoped disclosure, and notice — neither silent informant nor silent accomplice.