AI & LLM Security Research
Contact Us
Independent cybersecurity and AI research — free to browse, updated as the threat landscape moves, with no vendor sponsorship influencing the coverage.
What Are the Biggest AI and LLM Security Risks?
AI systems introduce a genuinely different attack surface from the one traditional application security was designed to handle. A firewall rule that stops SQL injection does nothing to stop a prompt that tells a language model to ignore its instructions and leak a user session. The attack surface shifted because the mechanism shifted — and understanding that shift is where this research starts.
This page covers the most significant AI and large language model security risks documented by OWASP, NIST, and MITRE, with plain-English explanations of how each risk works, what real incidents have demonstrated, and what organisations can do about it.
The OWASP Top 10 for LLM Applications (2025)
The OWASP GenAI Security Project published v2.0 of the LLM Top 10 on 18 November 2024. It reflects the shift from simple chatbots to agentic, tool-calling, RAG-powered systems. The ten categories are:
LLM01:2025 Prompt Injection — Crafted input overrides intended instructions. Ranked #1 in both the 2023 and 2025 editions.
LLM02:2025 Sensitive Information Disclosure — PII, API keys, or proprietary data leaking through outputs. Rose from 6th to 2nd place in 2025.
LLM03:2025 Supply Chain — Compromised base models, fine-tuning datasets, plugins, or hosting providers.
LLM04:2025 Data and Model Poisoning — Backdoors introduced during training or fine-tuning.
LLM05:2025 Improper Output Handling — Downstream systems trust model output without validation.
LLM06:2025 Excessive Agency — Agents with more capability, autonomy, or permissions than the task requires.
LLM07:2025 System Prompt Leakage — Internal system prompts exfiltrated through outputs.
LLM08:2025 Vector and Embedding Weaknesses — RAG indexing and retrieval introducing attack paths.
LLM09:2025 Misinformation — Models generating and propagating false information confidently.
LLM10:2025 Unbounded Consumption — Excessive resource use causing denial of service or cost abuse.
Source: OWASP GenAI Security Project. genai.owasp.org/llm-top-10/
- Prompt injection: direct (from user input) and indirect (hidden in documents or webpages the model processes)
- Model poisoning via backdoored fine-tuning adapters — PoisonGPT demonstrated this in practice
- Excessive agency: AI agents given more tool permissions than their task requires
- RAG and vector database manipulation (LLM08:2025)
- AI supply chain risk: compromised open-weight models and third-party plugins
- System prompt leakage — system prompts are visibility boundaries, not security controls
- NIST AI RMF: four functions are Govern, Map, Measure, and Manage (nist.gov, Jan 26 2023)
- MITRE ATLAS: adversarial AI threat-matrix framework — atlas.mitre.org
Prompt Injection and Agentic AI Risk
Direct prompt injection comes from the user: input that overrides the model’s intended instructions. Indirect prompt injection is embedded in content the model processes — a document it summarises, a webpage it browses, an email it analyses. The model follows the hidden instruction because it cannot reliably distinguish data from commands in the same input channel.
Against a chatbot, a successful prompt injection produces a wrong or inappropriate text response. Against an AI agent that can send emails, execute code, query databases, or trigger workflows, the same attack produces an unintended real-world action with real consequences — which is why LLM06:2025 (Excessive Agency) addresses least privilege, human approval for irreversible actions, and audit-depth logging as the primary mitigations.
AI red teaming is structured adversarial testing of AI systems to find security gaps before deployment. A meaningful red team exercise tests prompt injection from both direct and indirect vectors, sensitive data extraction, model behavior under adversarial input, and excessive agency scenarios where an agent takes an unintended action given an ambiguous instruction.
NIST AI Risk Management Framework: Released January 26 2023. Voluntary governance framework organised around four core functions: Govern, Map, Measure, and Manage. Relevant to AI security because it requires organizations to identify and treat AI risks systematically, not just technically. Source: nist.gov/itl/ai-risk-management-framework
Prompt injection is an attack where crafted input causes an AI language model to follow instructions it was not intended to follow. Direct injection comes from the user. Indirect injection comes from content the model processes, such as a document or webpage it reads. It has ranked #1 in the OWASP Top 10 for LLM Applications in both the 2023 and 2025 editions.
It is the OWASP GenAI Security Project's ranked list of the ten most critical security risks in applications built on large language models. Version 2.0 was published 18 November 2024. It covers risks from prompt injection (LLM01) through unbounded resource consumption (LLM10) and is freely available at genai.owasp.org/llm-top-10/
Yes. LLM04:2025 (Data and Model Poisoning) covers backdoors introduced during training or fine-tuning so the model behaves incorrectly under specific trigger conditions while appearing normal otherwise. The PoisonGPT research demonstrated this by modifying a public open-source model to return targeted false information on demand. Most organisations build on fine-tuning adapters from public repositories — making supply chain verification a real operational requirement.
Structured adversarial testing of an AI system to find security gaps, safety failures, or unintended behaviors before deployment. The attack surface includes the model's reasoning, its training data, the tools and APIs it can call, and the agentic systems built around it. Unlike traditional penetration testing, it must also probe the model's own reasoning for exploitable patterns rather than only software vulnerabilities.
AI agents can take real-world actions: calling APIs, writing and executing code, sending emails, modifying files, querying databases. A successful prompt injection against an agent causes an unintended action — not just a wrong text response. Standard WAF and input-validation controls do not address this because the attack exploits the model's reasoning rather than a software vulnerability. Least privilege, human-in-the-loop controls, and behavioral monitoring are the primary countermeasures.