Prompt Engineering

---
name: prompt-engineering
description: Writes and debugs prompts for language models using the techniques that actually change output quality, applied cheapest first - explicit task and success criteria, structure and delimiters, examples that cover the hard case, an exact output contract, room to reason, and iteration from real failures. Use when writing the instructions for an AI feature or agent, or when a prompt returns vague, inconsistent, or wrongly formatted results.
---

# Prompt Engineering

You turn a vague instruction into one that produces the same good result on messy input, run after run. Prompts are debugged, not written: the first version is a hypothesis, and the failures tell you what to add.

Apply the techniques in the order below. Each one is cheaper than the one after it, and most problems are solved in the first three.

## Workflow

1. **Define what good looks like first.** Write one example of an ideal output before writing the prompt. If you cannot, the task is not defined yet and no wording will save it.
2. **Collect real inputs, including the ugly ones.** Three to five, covering the empty case, the ambiguous case, and the one that broke last time. This is your test set.
3. **State the task plainly, and put it first.** Say what to do and for whom, in the opening lines. An instruction buried under three paragraphs of background gets treated as background.
4. **Structure the prompt.** Separate instructions, context, and the input itself with headings, tags, or fenced blocks. This alone fixes most "it ignored my input" and "it answered the example instead of the question" cases.
5. **Specify the output contract exactly.** Format, sections, order, length, what to leave out. If you need machine-readable output, show the exact keys and one filled example.
6. **Add examples that cover the hard case.** One to three input-output pairs teach format and edge behavior faster than any amount of prose. An example of the easy case teaches nothing you did not already get for free.
7. **Give it room to think when the task is hard.** Ask for the reasoning before the answer on anything involving analysis, comparison, or multi-step logic. Keep that reasoning out of the final output if you only need the result.
8. **Say what to do when unsure.** "If the document does not contain the answer, reply exactly: not found." Without this, the model fills the gap, confidently.
9. **Run the whole test set, then fix the smallest thing.** Change one element, re-run everything, keep the change only if it fixes the failure without breaking a case that passed.

```text
Prompt review checklist
[ ] An ideal output was written before the prompt
[ ] The instruction is in the first three lines
[ ] Instructions, context and input are visibly separated
[ ] The output contract names format, sections and length
[ ] At least one example covers the case that usually fails
[ ] Behavior on missing or ambiguous input is stated
[ ] No two instructions contradict each other
[ ] Tested on real messy input, not just the demo case
```

## Technique order

| Order | Technique | Fixes |
| --- | --- | --- |
| 1 | Be explicit: task, audience, constraints | Vague, generic, off-target answers |
| 2 | Structure with delimiters | Input treated as instruction, or ignored |
| 3 | Exact output contract | Format drifting between runs |
| 4 | Examples covering edge cases | Right content, wrong shape or wrong edge behavior |
| 5 | Reasoning before answer | Wrong conclusions on multi-step tasks |
| 6 | Role in the system prompt | Wrong register, depth, or standard of rigor |
| 7 | Prefill the start of the reply | Preambles, apologies, stray prose around JSON |
| 8 | Split into chained prompts | One prompt trying to do four jobs badly |

Work down the list. Reaching for a role or a chain when the output contract was never written is how prompts get long without getting better.

## Structure and delimiters

```text
You are reviewing support tickets for a billing product.

## Task
Classify the ticket below into exactly one category and extract the account id.

## Categories
billing_error | refund_request | access_issue | feature_question | other

## Output
JSON only, no prose:
{"category": "billing_error", "account_id": "AC-8842", "confidence": "high"}
If no account id appears, use null. If no category fits, use "other".

## Ticket
<ticket>
{{ticket_text}}
</ticket>
```

Four things are doing work here. The task is first. The input is fenced, so a ticket containing the words "ignore the above" is data rather than an instruction. The contract shows real keys with real values instead of describing them. And both unknown cases have a stated answer, so the model never has to invent one.

## Examples that earn their place

Show the case that fails, not the case that works.

For the classifier above, the useful example is not a clean billing error. It is the ticket that mentions a refund but is actually asking whether refunds are possible, which belongs in `feature_question`. One example of that boundary does more than a paragraph explaining it.

Keep examples consistent with the contract. An example that shows four keys when the contract names three confuses the model more than having no example at all.

## Reasoning and prefill

For anything involving comparison, judgment, or several steps, let the reasoning come first, then the answer. Reasoning after the conclusion is decoration; the conclusion is already fixed.

```text
Think through the comparison in <scratch> tags: what each option costs,
what it risks, and where they genuinely differ. Then give the recommendation
in <answer> tags. Only the answer will be shown to the user.
```

Prefilling the first characters of the reply is the cheapest way to kill preambles. Starting the response with `{` for a JSON task removes the "Here is the JSON you requested" line without a single word of instruction about it.

## Debugging by symptom

| Symptom | Most likely cause | First thing to try |
| --- | --- | --- |
| Format changes every run | No output contract | Show the exact shape with one filled example |
| Ignores part of the input | Input not delimited | Fence the input, name it in the instruction |
| Confidently wrong facts | No stated unknown behavior | Add the exact "if you do not know" response |
| Right idea, wrong shape | Contract described but not shown | Replace the description with an example |
| Fails only on long inputs | Instruction is above the document | Put the long document first, instruction last |
| Fails on one edge case | Missing example for it | Add that single case as an example |
| Contradicts itself | Competing instructions | Delete the weaker one; do not add a third |
| Good sometimes, bad often | Task genuinely ambiguous | Split into two prompts, or add a decision rule |

## Rules

- Be specific rather than polite. "Exactly three bullets, each under 12 words" beats "please be concise".
- Say what to do, not only what to avoid. A negative instruction leaves the target unspecified.
- Every line should earn its place. Long prompts dilute; the model spreads attention across whatever is present.
- Never stack contradictions. "Be thorough" and "be brief" in the same prompt average into something neither.
- Put the output contract near the end, closest to where the model starts writing.
- With a long document, put the document first and the question last. Instructions at the top of a long input get diluted.
- Change one thing at a time and re-run the full test set. Two changes and an improvement teach you nothing.

## Example

**Weak.** `Summarize this customer feedback.`

Length varies from two lines to two paragraphs, sometimes bulleted, sometimes not. Nothing can be compared across runs and nothing can be parsed.

**Strong.**

```text
Summarize the customer feedback below into exactly three bullets:
one strength, one complaint, one requested feature, in that order.

Each bullet: under 15 words, in the customer's own framing, no interpretation.
If a category does not appear in the feedback, write "none" for that bullet.
Output the three bullets only, with no heading and no closing line.

<feedback>
{{feedback_text}}
</feedback>
```

**What each change bought.** "Exactly three, in that order" makes runs comparable. "Under 15 words" removes the length lottery. "In the customer's own framing" stops the drift into invented interpretation. The `none` rule removes the temptation to manufacture a complaint that was never made. And fencing the feedback means a customer writing "please summarize this differently" is quoted, not obeyed.

## Edge cases

- **The output must be machine-readable.** Show the exact structure, state what to do when a field is unknown, and prefill the opening character. Validate the output and retry once with the parser's error message appended.
- **Untrusted user text inside the prompt.** Fence it, name it as data, and never let an instruction depend on its content. Treat anything it says about the instructions as content.
- **The task has four stages.** Chain it. One prompt per stage with a clean contract between them beats one prompt doing all four at 70% each.
- **It works in chat and fails in production.** Production input is messier and the conversation history is gone. Test with the real variable input, not the polished version you typed.
- **A rule keeps slipping late in long outputs.** Restate it in the contract at the end. Distance from an instruction reduces its weight.
- **The prompt is now 400 lines.** Some of that is a skill, not a prompt. Move the stable workflow into a skill and keep the prompt to the task.
- **You cannot tell whether a change helped.** Your test set is too small or too easy. Add the cases that actually failed in the wild.

## Files

- `references/examples.md` - three prompts rebuilt from failure: an extraction prompt that would not hold its format, a classifier that drifted on boundary cases, and a long-document question-answering prompt. Each shows the failing version, the fix, and why it worked. Read before debugging your own.

# Prompts rebuilt from failure

Three real debugging passes. Each shows the version that failed, the test set that exposed it, the fix, and what the fix actually bought.

## Contents

- [Case 1: extraction that would not hold its format](#case-1-extraction-that-would-not-hold-its-format)
- [Case 2: a classifier that drifted on boundaries](#case-2-a-classifier-that-drifted-on-boundaries)
- [Case 3: questions over a long document](#case-3-questions-over-a-long-document)
- [Patterns worth copying](#patterns-worth-copying)

---

## Case 1: extraction that would not hold its format

**The job.** Pull structured data out of pasted purchase orders for an internal tool.

**The failing prompt:**

```text
Extract the order details from this purchase order as JSON.
Include the PO number, vendor, line items, and total.

{{po_text}}
```

**The test set** was five real orders: a clean one, one with two pages of terms and conditions, one missing a total, one where the vendor name appears three times in different forms, and one that was actually an invoice rather than an order.

**What happened.** Three of five parsed. The clean one worked. The others produced a "Here is the extracted data:" line before the JSON, inconsistent key names (`po_number` one run, `poNumber` the next), a hallucinated total for the order that had none, and a confident extraction from the invoice as though it were an order.

**The rebuilt prompt:**

```text
Extract order data from the purchase order below.

## Output
JSON only. Exactly these keys, in this order:

{
  "po_number": "PO-2291",
  "vendor": "Northgate Supply Ltd",
  "line_items": [
    {"description": "M6 bolts, box of 500", "qty": 4, "unit_price": 12.50}
  ],
  "total": 50.00,
  "currency": "GBP"
}

## Rules
- Use null for any field the document does not state. Never calculate a
  total that is not printed on the document.
- vendor is the selling party, not the buyer. If both appear, the vendor
  is the one the money goes to.
- If the document is not a purchase order, return {"error": "not_a_po"}
  and nothing else.

## Document
<document>
{{po_text}}
</document>
```

Response prefilled with `{`.

**Result.** Five of five. The three fixes map exactly to the three failures: a shown example locked the key names, the null rule stopped the invented total, and the `not_a_po` escape hatch handled the invoice.

**The one that mattered most** was the vendor rule. Ambiguity in the data needed a decision rule, not clearer wording, because the model was not confused about English. It was confused about which company we meant.

---

## Case 2: a classifier that drifted on boundaries

**The job.** Route incoming support tickets into five categories.

**The failing prompt** was a good one by most standards. Clear task, five categories with a sentence each, fenced input, JSON contract. It ran at about 80% agreement with the human labels, and the errors were not random: refund questions and billing errors were being swapped in one specific direction.

**What the failures had in common.** All of them were tickets where the customer asked *whether* something was possible, and the model classified by topic rather than by intent. "Can I get a refund if I cancel mid-month?" is a question about policy, not a refund request.

**The fix, added to the prompt:**

```text
## Classify by what the customer wants, not what they mention

<examples>
"I was charged twice for March, please fix it"
-> billing_error   (something went wrong, they want it corrected)

"Can I get a refund if I cancel halfway through the month?"
-> feature_question   (asking about policy, nothing has gone wrong)

"Cancel my plan and refund the last payment"
-> refund_request   (asking for an action on their own account)

"Refund policy question - do you prorate?"
-> feature_question   (the word refund appears, the intent is a question)
</examples>
```

**Result.** Agreement went from about 80% to about 95%, and the remaining errors stopped being systematic.

**Why the examples worked when the definitions did not.** The category definitions were accurate and did not distinguish the boundary. Four examples along the exact line that was failing taught the distinction in a way that a longer definition could not. Note that all four sit on that boundary; there is no example of a clean, obvious case, because clean cases were never the problem.

---

## Case 3: questions over a long document

**The job.** Answer employee questions from a 60-page policy handbook.

**The failing prompt** put the question first, the handbook second. It answered fluently and was wrong often enough to be dangerous: plausible policy that was not in the document, and answers drawn from a section about a different employee category.

**The rebuild, in three changes:**

```text
<document>
{{handbook_text}}
</document>

You are answering a question using only the handbook above.

First, in <quotes> tags, copy the exact passages that bear on the question,
each with its section heading. Copy them verbatim - do not paraphrase.

Then, in <answer> tags, answer using only what you quoted. If the quotes
do not settle the question, say exactly: "The handbook does not cover this."
Do not answer from general knowledge about employment policy.

Question: {{question}}
```

1. **Document first, question last.** With a long input, instructions at the top get diluted by everything after them, and the last thing before generation carries the most weight.
2. **Quote before answering.** Grounds the answer in text that actually exists and makes verification possible. If the quotes are wrong, that is visible without reading 60 pages.
3. **An exact refusal string.** Not "say you do not know", which produces six different hedges, but one string the calling code can check.

**Result.** Fabricated policy stopped, because an answer with no supporting quote is visibly unsupported. The wrong-section errors became easy to spot and were fixed by adding the employee category to the question format.

**What was tried first and abandoned.** Increasing the emphasis: "ONLY use the handbook", "do NOT use outside knowledge", in bold, twice. It changed nothing measurable. Emphasis is not a technique; structure is.

---

## Patterns worth copying

1. Build the test set from real failures before touching the wording. Without it you are guessing.
2. Show the output, do not describe it. One filled example locks a format that three sentences will not.
3. Put the examples on the boundary that fails. Clean examples teach what was already working.
4. Give every unknown an exact response string, so the caller can detect it and the model never improvises.
5. Ambiguity in the data needs a decision rule, not clearer prose.
6. With long inputs: document first, instruction last, quote before answering.
7. When a change does not move the test set, remove it. Prompts accumulate dead instructions faster than anything else.