AI Security & Safety
Attacks on AI systems and how to defend them: prompt injection, red teaming, and the frameworks security teams use.
Beginner
Start here. No background needed beyond school maths.
Gandalf: Test Your Prompt-Injection Skills
A browser game where you trick an AI into revealing a secret password across levels of rising defence. A safe way to feel how prompt injection works.
Sign up to see the source
OWASP Top 10 for Large Language Model Applications
The ten biggest security risks in LLM apps, including prompt injection, leaking sensitive data and insecure plugins, each with examples and ways to defend.
Sign up to see the source
Simon Willison: Prompt Injection
A running series of clear write-ups on prompt injection: what it is, why it is hard to fix and why tool-using agents make it more dangerous.
Sign up to see the source
Advanced
Once the basics are comfortable.
garak: The LLM Vulnerability Scanner
Scan a language model for weak spots such as prompt injection, jailbreaks, data leakage and toxicity, much like a network scanner checks a server.
Sign up to see the source
Google Secure AI Framework (SAIF)
••••'s framework of practical controls for securing AI systems across data, models, deployment and monitoring, with a risk map to match.
Sign up to see the source
Microsoft AI Red Team
How •••• tests AI systems by attacking them first: the approach, the lessons, and the tools and guidance it shares with others.
Sign up to see the source
NIST: Adversarial Machine Learning Taxonomy
A government standard that names and organises attacks on AI (evasion, poisoning, privacy and abuse attacks) and the defences for each. Useful as a reference list.
Sign up to see the source
PyRIT: Python Risk Identification Tool for Generative AI
An open-source toolkit for red-teaming generative AI: automate attack prompts against a model, score the answers and keep records of what you tried.
Sign up to see the source