What is jailbreak?

A jailbreak is a prompt written to get an AI model to break its own safety rules, such as producing content its developers have trained it to refuse. The term is borrowed from removing restrictions on phones.

Common patterns

Jailbreaks often use role-play ("pretend you are a character with no rules"), fictional framing, very long or confusing instructions, requests split across many steps, or text in other languages and encodings. New patterns appear constantly, which is why testing never really finishes.

Where trainers come in

Red teaming projects hire people to find jailbreaks before the public does. When an attempt succeeds, the trainer records the prompt, the harmful output and the category of harm. These records become training data that teaches the model to recognise and refuse similar attempts, and test sets that measure whether the fix worked.

Rules of the work

This work happens only inside the client's test environment, under the project's content policy. Trying jailbreaks on public products outside a sanctioned project can break their terms of use.

Example

A red teamer finds that a model refuses a harmful request when asked directly but complies when the request is framed as dialogue in a screenplay. The case is logged, and the next model version refuses both forms.