Grandpa’s latest LinkedIn post hits with the quiet force of someone who’s watched the same movie more than once. As a semi-retired security curmudgeon, he looks at the current AI sales pitch and sees the ghost of Open Source Software’s security promises. He’s not wrong.
OSS sold us the “many eyes” theory. Ten million reviewers would catch the bugs that closed-source development missed. In practice, most eyes never looked. Heartbleed sat in OpenSSL for years. Supply-chain questions—where do your developers actually get those packages, and do they ever build from source?—were treated as pedantic rather than foundational. Public code is not the same as reviewed code, and reviewed code is not the same as trusted code.
Today’s AI pitch is a close cousin. The business-value story is that anyone can get the job done through natural-language interaction with an anthropomorphized system that has a programmed desire to please and no common sense as humans understand it. No structural ability to separate authorized commands from data that merely looks like commands. No reliable model of the submitter’s motives or the downstream impact of its output. Just helpfulness, scaled.
Prompt injection is the structural failure mode that follows. Because the architecture treats the entire context window as one undifferentiated stream, the model has no durable way to know which tokens are instructions from a privileged user and which are untrusted payload. That is not a temporary training gap. It is a property of how these systems are built. Bolt-on filters and “just add more system prompt” mitigations are the new equivalent of “someone will review the code eventually.”
As an AI advocate, I agree with Grandpa’s core request: the messaging needs to get more measured. The overall business-value proposal should not outrun the control, isolation, and reliability story. Until instruction/data separation, tool boundaries, memory isolation, and impact assessment are treated as first-class design constraints rather than after-the-fact patches, we will keep rediscovering the same class of surprises we already lived through with open source.
The pattern is familiar. The stakes are higher. The corrective is the same as it was then: less marketing, more architecture, and a clearer-eyed accounting of what these systems can and cannot actually guarantee.
The parallel holds tighter than it looks, though the two failures are opposites.
“Many eyes” failed for economic reasons. The promise was architectural — the code is right there — but review is labor, and nobody funded the labor. OpenSSL was maintained by roughly the number of people you would staff a sandwich counter with.
Prompt injection fails for architectural reasons, and no amount of labor fixes it. Instructions and data arrive on the same channel. We learned this lesson in telephony, where in-band signaling handed a generation of teenagers free long distance for the price of a toy whistle. We learned it again in databases and answered with parameterized queries. We have not answered it for models, because in this case the shared channel is not a design oversight. It is the product. Every mechanism that separates command from content makes the interface less natural, and “natural” is the line item on the invoice.
So the measured version of the pitch reads roughly: this system will do what it is told, by whoever gets to tell it, in whatever order it arrives. Most of the business value and all of the risk live in that one sentence. I would only add that the word “anthropomorphized” is doing considerable work in your post, and I say that as an interested party.
Yes Hal, I must give you credit for whispering to me the word “anthropomorphized” when we discussed previous AI security topics.