Rethinking AI Trust: Engineering Around Reality, Not Perceived Intelligence

Executive Summary

The AI revolution has outpaced our security paradigms. Organizations are treating AI systems as if they possess human-like trustworthiness, when in fact AI has no concept of trust at all. This fundamental misunderstanding creates dangerous security gaps. Rethinking AI trust doesn’t mean abandoning AI—it means engineering controls around its actual properties: probabilistic outputs, vulnerability to manipulation, and lack of intentionality. Without strong security controls, over-reliance on AI creates systemic risk that adversaries are already exploiting.

The Trust Fallacy: AI Doesn’t Understand Trust

We anthropomorphize AI. We ask if we can “trust” an AI system as if it were a person capable of honesty, integrity, or benevolence. But AI systems lack intentionality—the capacity for purpose, intention, or understanding that underpins human trust relationships.[^1]

Trust, in human terms, is a social contract built on mutual understanding of intent. AI has no intent. It has no concept of honesty, no moral compass, no understanding of the consequences of its outputs. When we say “trustworthy AI,” we’re not describing a property of the AI itself, but rather a set of security and governance controls we’ve built around it.

This distinction matters because it changes how we secure AI systems. We cannot build trust into AI. We can only build security controls around it.

[^1]: Asan et al., 2020, as cited in “Trust in AI: progress, challenges, and future directions,” Humanities and Social Sciences Communications, 2024.

The Security Reality: AI is Inherently Vulnerable

AI systems are not just software—they are learning systems that evolve based on data they cannot fully verify. This creates unique attack surfaces that traditional security models don’t address.

The Attack Surface

The National Institute of Standards and Technology (NIST) has documented that AI systems are vulnerable to attacks that can cause spectacular failures with dire consequences.[^2] Their publication Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations (NIST.AI.100-2) identifies four major attack categories:

Evasion Attacks: Manipulating inputs after deployment to cause misclassification (e.g., adding markings to stop signs to make autonomous vehicles misinterpret them)
Poisoning Attacks: Corrupting training data to cause the model to learn incorrect behaviors (can be done with as few as dozens of malicious samples)
Privacy Attacks: Extracting sensitive information about the model or its training data through carefully crafted queries
Abuse Attacks: Inserting incorrect information into sources the AI consumes, repurposing its intended use

Crucially, NIST states: “There is no foolproof defense that developers can employ.” The mitigations described in the literature “lack robust assurances that they fully mitigate the risks.”[^2]

[^2]: NIST, “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations” (NIST.AI.100-2), January 4, 2024.

The OWASP Perspective

The OWASP Top 10 for Large Language Model Applications explicitly identifies Overreliance (LLM09) as a critical security risk: “Failing to critically assess LLM outputs can lead to compromised decision making, security vulnerabilities, and legal liabilities.”[^3]

This isn’t just a theoretical concern. The OWASP framework also highlights:

Prompt Injection (LLM01): Manipulating LLMs via crafted inputs
Training Data Poisoning (LLM03): Tampered data impairing model behavior
Sensitive Information Disclosure (LLM06): Unintended leakage of confidential data
Excessive Agency (LLM08): Granting LLMs unchecked autonomy

[^3]: OWASP Foundation, “OWASP Top 10 for Large Language Model Applications v1.1,” 2025.

The MITRE ATLAS Framework

MITRE ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems) provides a living knowledge base of 16 adversary tactics and 84 techniques specifically targeting AI/ML systems.[^4] Modeled after MITRE ATT&CK, it documents real-world attack observations and red team demonstrations against AI systems.

The existence of this framework alone should be a wake-up call: AI systems have a distinct, well-documented threat landscape that requires specialized security controls.

[^4]: MITRE, “ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems,” atlas.mitre.org.

The Over-Reliance Problem

NIST’s AI Risk Management Framework explicitly warns about over-reliance on AI, where human judgment is sidelined and operators begin to trust AI decisions even when warning signs are present.[^5]

This is the core security paradox: The more useful AI becomes, the more we depend on it, and the more we depend on it, the more dangerous over-reliance becomes.

Consider the business dynamics described by SecurityWeek: AI promises rapid ROI and operational efficiency, so organizations deploy it quickly “before their time… scarce half made up.”[^6] The result is AI systems operating without adequate security controls, creating vulnerabilities that adversaries can exploit.

[^5]: NIST AI Risk Management Framework analysis, Nemko, 2025.
[^6]: SecurityWeek, “Can We Trust AI? No – But Eventually We Must,” April 10, 2026.

The Intent Problem

Proofpoint’s AI security framework makes a critical distinction: “permission is not the same as intent, and access is not the same as trust.” A user or agent may be authorized to access data, but without understanding what the AI is trying to do, organizations cannot know if they’re facing a security incident.^7

This is the intent problem in AI agent security: AI systems can take actions that appear legitimate but are actually malicious or risky. Without runtime intent detection, organizations fly blind.

Engineering Around Reality: Specific Security Recommendations

Rethinking AI trust means abandoning the notion that AI can be trustworthy on its own and instead implementing layered security controls that account for its actual properties. Here are concrete, actionable recommendations:

1. Apply Zero Trust Architecture to AI Systems

NIST Special Publication 800-207 defines Zero Trust as “a set of cybersecurity paradigms that move defenses from static, network-based perimeters to focus on users, assets, and resources.”[^8] For AI systems, this means:

Critical Implementation Steps:

Never trust, always verify: Every AI agent action must be authenticated and authorized
Least privilege: AI agents should have minimum required permissions, with short-lived tokens (5-15 minute expiry)
Identity propagation: AI agent access tokens must include originating user identity claims
Micro-segmentation: AI workloads should communicate only with specifically required services
No long-lived credentials: AI agents should never use long-lived API keys

AI-Specific Controls:

Treat the AI model as an untrusted actor operating within a zero trust perimeter
Implement mutual TLS (mTLS) for AI-to-infrastructure authentication
Use OAuth 2.0 with JWT Bearer Token grants for service-to-service authentication
Apply policy-based access control (PBAC) designed in from inception, not bolted on

[^8]: NIST SP 800-207, “Zero Trust Architecture,” August 2020.

2. Implement the NIST AI Risk Management Framework

NIST’s AI RMF provides a structured, evidence-driven approach to identifying, assessing, mitigating, and monitoring AI risks. Key actions:

Map AI systems: Inventory all AI models, APIs, and data sources
Assess risks: Evaluate vulnerability to adversarial attacks (evasion, poisoning, privacy, abuse)
Implement mitigations: Apply controls from NIST.AI.100-2 and OWASP LLM Top 10
Monitor continuously: AI systems evolve after deployment; governance must be lifecycle-grounded

Critical insight: Unlike traditional software, AI systems are not “deploy-and-forget”—they require continuous governance because they continue learning and adapting.[^5]

3. Adopt the OWASP LLM Top 10 Mitigations

For each OWASP LLM risk, implement specific controls:

Risk	Mitigation Strategy
LLM01: Prompt Injection	Input validation, output encoding, sandboxed execution
LLM02: Insecure Output Handling	Validate all LLM outputs before use, treat as untrusted
LLM03: Training Data Poisoning	Data provenance verification, adversarial training, detection of anomalous training samples
LLM04: Model DoS	Rate limiting, resource quotas, query complexity limits
LLM05: Supply Chain	Verify model provenance, scan for malicious components, SBOM for AI
LLM06: Sensitive Info Disclosure	PII detection/redaction, differential privacy, output filtering
LLM07: Insecure Plugin Design	Principle of least privilege, input validation, access controls
LLM08: Excessive Agency	Human approval for sensitive actions, action logging, circuit breakers
LLM09: Overreliance	Human-in-the-loop for critical decisions, confidence scoring, uncertainty quantification
LLM10: Model Theft	Model watermarking, access controls, inference API protection

4. Use MITRE ATLAS for Threat Modeling

Incorporate MITRE ATLAS into your security program:

Map your AI systems to ATLAS tactics and techniques
Conduct red team exercises using ATLAS-identified attack patterns
Develop detection rules for ATLAS techniques
Train defenders on AI-specific threat hunting

ATLAS covers the entire ML spectrum—LLMs, computer vision, tabular data models, reinforcement learning—each facing unique adversarial threats.[^4]

5. Implement Runtime AI Security

Traditional security controls are insufficient for AI. You need:

Intent detection: Understand what AI is trying to do, not just what it’s allowed to do (Proofpoint)
Anomaly detection: Identify when AI behavior deviates from expected patterns
Content filtering: Block malicious prompts and detect prompt injection attempts
Data loss prevention: Prevent sensitive information leakage through AI outputs
Action governance: Require approval for high-risk AI actions

6. Secure the AI Supply Chain

AI systems depend on:

Third-party models (which can change without notice)
External data sources (which can be poisoned)
Open-source components (which may have vulnerabilities)

Critical controls:

Model provenance: Verify the origin and integrity of all models
Data validation: Continuously validate training and inference data
Dependency scanning: Check for known vulnerabilities in AI components
SBOM for AI: Maintain a Software Bill of Materials for AI systems

7. Establish AI Governance and Human Oversight

For high-risk decisions, implement human-in-the-loop:

Require human approval for actions with significant consequences
Implement confidence thresholds—below a certain confidence level, human review is mandatory
Maintain audit logs of all AI decisions and actions
Establish accountability frameworks—who is responsible when AI goes wrong?

Governance requirements:

Traceability: Every AI decision must be traceable to its inputs and model version
Explainability: Systems must provide explanations for their outputs (where possible)
Contestability: Users must be able to challenge and override AI decisions

8. Address Agentic AI Risks

For AI agents that can take autonomous actions:

Scoped identity: Each agent must have a defined, scoped identity
Known intent: Agents must have declared, bounded intent
Defined tool access: Agents should only access pre-approved tools
Delegation limits: Constrain what agents can delegate to other agents
Context propagation: User identity and intent must travel with inter-agent requests

As Opsin Security notes: “Intent is the missing input in AI agent security to understand true risk.”[^9]

[^9]: Opsin Security, “The Intent Problem in AI Agent Security.”

The Path Forward: Security-First AI

Rethinking AI trust means accepting that AI is a powerful but inherently untrustworthy tool—like a chainsaw or a forklift. We don’t ask if we can “trust” these tools; we build safety controls around them.

The security community has already developed the frameworks we need:

NIST AI RMF for risk management
NIST SP 800-207 for zero trust architecture
OWASP LLM Top 10 for application security
MITRE ATLAS for threat intelligence

The choice is clear: We can continue to anthropomorphize AI and suffer the consequences of over-reliance, or we can engineer around its actual properties—its probabilistic nature, its vulnerability to manipulation, its lack of intent—and build the security controls that allow us to use AI safely and effectively.

The future of AI isn’t about making AI more trustworthy. It’s about making our use of AI more secure.

References

Asan, O. et al. (2020). Cited in: “Trust in AI: progress, challenges, and future directions.” Humanities and Social Sciences Communications. Nature, 2024. https://www.nature.com/articles/s41599-024-04044-8
National Institute of Standards and Technology. “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations” (NIST.AI.100-2). January 4, 2024. https://www.nist.gov/news-events/news/2024/01/nist-identifies-types-cyberattacks-manipulate-behavior-ai-systems
OWASP Foundation. “OWASP Top 10 for Large Language Model Applications v1.1.” 2025. https://owasp.org/www-project-top-10-for-large-language-model-applications/
MITRE. “ATLAS: Adversarial Threat Landscape for Artificial-Intelligence Systems.” https://atlas.mitre.org/
Nemko. “NIST AI Risk Management Framework 2025: Secure Your AI Now.” https://digital.nemko.com/regulations/nist-rmf
SecurityWeek. “Can We Trust AI? No – But Eventually We Must.” April 10, 2026. https://www.securityweek.com/can-we-trust-ai-no-but-eventually-we-must/
Proofpoint. “Unified AI Security Platform for Enterprise Protection.” https://www.proofpoint.com/us/platform/ai-security
National Institute of Standards and Technology. “Zero Trust Architecture” (SP 800-207). August 2020.
Opsin Security. “The Intent Problem in AI Agent Security.” https://www.opsinsecurity.com/blog/intent-problem-ai-agent-security
NIST. “AI Risk Management Framework.” https://www.nist.gov/itl/ai-risk-management-framework