A follow-up to Why AI Should Be Treated as an Untrusted Component
In the last article, I argued that AI systems belong in the same threat model as any other untrusted component—user input, third-party APIs, unverified dependencies. The reaction I heard most often, informally, was some version of: sure, but it doesn’t feel like a dependency. It feels like it’s thinking.
That feeling is the vulnerability. Not a metaphorical one—an operational one, with a CVE-shaped hole in it. I want to pull that thread on its own this time, because anthropomorphization isn’t a harmless communication shortcut layered on top of the untrusted-component problem. It’s a mechanism that actively degrades security posture, and it does so in specific, traceable ways.
Trust Is a Budget, and Language Spends It
Security engineering runs on a simple discipline: every privilege, every exception, every “just this once” has to be justified and logged. Trust is not a feeling you extend generously—it’s a budget you allocate deliberately, and every dollar you don’t account for is a dollar an attacker can find.
The words we use for AI systems spend that budget without anyone noticing. When a system “understands” a request, “believes” a fact, “decides” on a course of action, or “wants” to help, we’re borrowing vocabulary built for entities with intent, memory, accountability, and stakes. None of those properties are actually present. What’s present is a statistical process generating the next most plausible token given a context window. But the vocabulary implies a mind, and minds get budgeted trust that architectures don’t.
This isn’t pedantry about word choice. It’s the difference between an engineer asking “what happens if this component’s output is wrong or adversarial” and an engineer asking “what happens if this colleague makes a mistake.” The first question produces threat models, sandboxes, and validation layers. The second produces the same social trust calculus we use for people—give them the benefit of the doubt, assume good faith, escalate only after repeated failures. That calculus is exactly wrong for a system that has no faith to be good, and that can be manipulated by anyone who can craft the right input.
The Confidence Trick Has No Trickster
Human confidence is a signal we’ve spent our whole lives learning to calibrate. We read hedging, tone, body language, track records, and reputation to decide how much to trust what someone tells us. Con artists exploit this calibration deliberately—they perform confidence they haven’t earned. It works because we assume confidence usually correlates with actual certainty, because for humans, most of the time, it does.
Large language models produce fluent, confident output regardless of whether the underlying claim is correct. There is no trickster behind this confidence—no intent to deceive, no awareness that a deception is even occurring. That absence of intent makes it harder to defend against, not easier, because none of the social heuristics we use to detect deception apply. There’s no nervous tell, no story that doesn’t quite add up under questioning, no incentive structure to reason about. The model isn’t lying to you. It’s also not telling the truth. It’s doing neither, and the fluent packaging makes that distinction nearly invisible to the person on the receiving end.
Anthropomorphizing the system encourages exactly the wrong response to this: it invites us to apply human-deception detection, which finds nothing wrong, and to conclude the output must therefore be trustworthy. The absence of a lie gets read as the presence of truth. Neither is warranted.
“It’s Just Being Helpful” Is Not a Security Control
Prompt injection is the clearest place this failure shows up operationally. When an AI agent reads a document, browses a page, or processes an email that contains embedded instructions, and then follows those instructions against the user’s actual intent, the postmortem language matters enormously. Framed as “the model was tricked” or “the model got confused,” the incident reads like a lapse in judgment—something you’d coach a person through. Framed as “the system executed attacker-controlled input because input and instruction were never separated,” the incident reads like what it is: a missing security boundary.
Only one of those framings tells you what to fix. Coaching a statistical process to be less “gullible” is not an engineering intervention—there’s no judgment in there to sharpen. Separating instruction channels from data channels, sandboxing tool access, and scoping credentials are engineering interventions. Anthropomorphized language steers postmortems toward the framing that produces no fix.
The same distortion shows up in access decisions before an incident ever happens. Teams that think of their AI agent as “a very capable new hire” tend to provision it the way they’d provision a new hire: broad enough access to be useful, with the assumption that judgment and accountability will keep it in bounds. But the agent has no judgment to keep it in bounds—it has a context window and a policy that can be overridden by whoever controls what lands in that context window. A new hire who did something harmful because a stranger’s email told them to would face consequences that create a feedback loop discouraging repetition. The model has no such loop. It will do it again the next time the input looks similar, with the same fluent confidence.
Where the Language Comes From, and Why It’s Sticky
None of this is really an accident. The industry’s own vocabulary—”understanding,” “reasoning,” “hallucination” instead of “unfaithful output,” even “intelligence” in the product category name—was chosen because it’s intuitive, marketable, and genuinely useful shorthand for describing capability. It’s not wrong that these systems produce outputs that look like the products of understanding. The shorthand is doing real communicative work, and I’m not going to pretend a security newsletter is going to retire it.
But shorthand designed for marketing and product intuition is not the same vocabulary you want running underneath a threat model. The two audiences need different words for the same behavior, and right now they’re using the same words for both, which means the marketing framing—optimized to build comfort and adoption—leaks directly into the engineering framing, which needs the opposite: discomfort and verification.
What Changes When You De-Anthropomorphize the Threat Model
The fix isn’t a company-wide ban on the word “understands.” It’s making sure the people who write access policies, design agent architectures, and run incident postmortems are working from mechanical language, even if the product marketing keeps the friendlier version.
A few concrete shifts follow from that:
- Access provisioning should be scoped to what the system’s outputs can be verified to need, not to what a “helpful colleague” would plausibly require. Least privilege doesn’t bend for good intentions the system doesn’t have.
- Incident postmortems should describe failure in terms of missing boundaries and unvalidated inputs, not “confusion” or being “tricked.” If the write-up would read the same for a person, rewrite it.
- Escalation logic should never rely on the system recognizing its own uncertainty and asking for help the way a cautious employee would. It has no reliable internal signal for that, and treating fluent output as a proxy for confidence is the exact trap.
- Procurement and vendor evaluation should ask what mechanical guarantees exist—sandboxing, input/output validation, provenance—rather than accepting “it’s very advanced” as a substitute for evidence.
None of this requires believing AI is unimpressive. It requires believing that impressiveness and trustworthiness are different axes, and that the words we’ve inherited for describing the first one keep sliding, unearned, onto the second. Security engineering has never extended trust based on how something feels to talk to. The moment we start, for AI, is the moment we’ve quietly repealed a discipline that took the rest of the industry decades to build.
Treat the language as carefully as you treat the access token. Both are attack surface if you’re not paying attention.
