Back in August, we took Grandpa’s LinkedIn post seriously and argued that the current AI sales pitch rhymes with open source’s old security promises. (7312.us) Seven weeks later, it is worth asking whether anything has changed — and the honest answer is: the rhyme has settled into a refrain.
What the original post got right
The core claim still stands. “Many eyes” failed because public code is not the same as reviewed code, and reviewed code is not the same as trusted code. Heartbleed sat in OpenSSL for years while everyone assumed someone else was looking. Today’s AI pitch repeats the structure: a business-value story that runs ahead of the control story, with prompt injection as the built-in failure mode that follows. The context window remains one undifferentiated stream; the model still has no durable way to separate privileged instructions from untrusted payload. Bolt-on filters are still the new “someone will review the code eventually.”
The comment thread added the sharpest point of all: the two failures are opposites. Many-eyes failed for economic reasons — the code was right there, but review is labor and nobody funded the labor. Prompt injection fails for architectural reasons, and no amount of labor fixes it. In-band signaling gave a generation of phone phreakers free long distance; SQL injection gave us parameterized queries; the LLM context window gives us… nothing yet, because the shared channel is not an oversight — it is the product.
What has happened since
- The marketing has not gotten more measured. If anything, the anthropomorphized “helpful assistant” framing has intensified. Agentic features continue to ship with tool access, memory, and browser integration that widen the blast radius of exactly the failure mode we described.
- The corrective vocabulary has matured. Instruction/data separation, tool boundaries, memory isolation, and impact assessment are no longer fringe vocabulary — they are increasingly what enterprise buyers ask about in diligence. The demand side of the market is learning faster than the supply side is conceding.
- The pattern keeps confirming itself. Every few weeks brings another demonstration that a system prompt is not a security boundary. We are, predictably, rediscovering the class of surprises we already lived through with open source — at higher stakes.
The measured pitch, one more time
The honest business-value sentence remains the one from the comments: this system will do what it is told, by whoever gets to tell it, in whatever order it arrives. Most of the business value and all of the risk live in that one sentence.
Until instruction/data separation is a first-class design constraint rather than an after-the-fact patch, the hype cycle will keep rhyming. The corrective is the same as it was then: less marketing, more architecture, and a clearer-eyed accounting of what these systems can and cannot actually guarantee.
The pattern is familiar. The stakes are higher. The song is the same — we are just further into the second verse.
