← Back to field notes
No. 08 Field note

What Actually Works in System Prompting

Four things that have held up across the agents I've built, and the bit of model mechanics they all fall out of. A system prompt isn't a settings page you configure, it's the front of a sequence the model reads back one token at a time, and most of what's worth knowing about prompting is a consequence of that.

30 June 2026 · Fundamentals · agents · prompting · 9 min read

Prompting is the craft of writing clear, effective instructions for an agent to work from. It’s a wide subject, so this isn’t a survey: it’s the handful of things that have held up across the agents I’ve actually built, in the order I go through them.

One thing to say up front, because it shapes everything below. Most of the work in a good system prompt is deciding what doesn’t go in it. In the last piece I listed instructions as one component of a harness, sitting alongside tools, memory, control and everything else. This is what’s inside that box, and it’s smaller than you’d expect.

First, how the model reads what you wrote

Before any of the techniques, it’s worth being clear about how a language model actually produces text, because most of what follows falls out of it.

An LLM is autoregressive. It generates one token at a time, and every token it produces is conditioned on all the tokens that came before it, including the ones it just wrote itself. There’s no separate instruction layer sitting above the text. Your system prompt isn’t a setting you configure, it’s the opening stretch of one long sequence the model reads back on every token it generates. I covered the consequences of this for reasoning in the chain of thought piece; here it matters for a more mundane reason.

Length is a cost rather than a free upgrade, because everything in the window competes for the same attention. And the model can’t tell your sentences apart from the ones a tool handed back, so tool descriptions and tool output land as instructions just the same, whether you wrote them or not. How an LLM works with regards to attention is a large topic in its own right, and I’m not going to cover it properly here.

Evals come before the agent, not after it

The first thing on the list isn’t a prompting technique at all. Before I build any part of an agent now, I get the evaluations in place: a fixed set of cases the agent has to handle and a way of scoring how it did.

The reason is what a system prompt actually is. It’s not a thing you write once and walk away from, it’s a thing you engineer. You’ll iterate on it, and every iteration is a claim that the agent is better than it was a version ago. Without evals you can’t check that claim, so you’re not improving a system prompt, you’re just changing one. You move a line, the next run looks better, and you have no way of knowing whether that was the change or the sampling. Then you make ten more changes on the same basis and the system prompt slowly fills up with edits that never did anything. Obsess over the cleanliness of the system prompt: no filler, every line earning its place as the highest-signal thing you could have put there, and evals are the only way to know which lines those are.

The order matters more than people expect. Add the evals after the agent works and every decision you took getting there was taken on feel, including the ones you’d now struggle to justify. Put them in first and system prompt work turns into something you can actually run: change one thing, run the cases, keep it or throw it away.

Start with the smallest system prompt that could work

With the evals in place, the first version of the system prompt should be close to bare. Four things: the role, the data or documents the agent is working from, the task and what done looks like, and the shape of the output. That’s it. Everything beyond those four is something you should have to earn, and the evals are what let you earn it rather than guess at it.

Role is the one that’s changed most. We all used to open system prompts with a line about being a world-class this or a senior that, and it did move the output, so it stuck around as a habit long after it needed to. Models have got a lot better since and that line doesn’t buy much any more. Role hasn’t stopped mattering, it’s stopped being free performance.

Where it still earns its place is when you want the agent pointed at something narrow, a particular discipline or a particular way of working a problem. Worth knowing that this cuts both ways though. A tight role doesn’t just aim the agent, it fences it in. Tell it it’s a database specialist and it’ll get oddly cautious about anything that isn’t a database, including things you’d have been happy for it to handle. That’s useful when you want the fence and irritating when you don’t, so pick the role for the constraint it puts on the agent rather than as a compliment.

Structure has moved in the same direction as role. XML tags were the convention for a long while and there was a good reason for them, but the newer models don’t need the tagging to work out where one part of your system prompt ends and the next begins. Plain markdown headers do the job now, and they have the advantage of being readable by the person maintaining the system prompt as well as the model reading it.

## Role
You triage incoming support tickets for a B2B software product.

## Documents
The ticket, the customer's plan and contact history, and the product docs.

## Task
Work out what the customer is actually asking for and answer it from the docs.
Where the docs don't settle it, say so rather than filling the gap.

## Done
The customer has an answer they can act on, or the ticket is with a human.

## Output
JSON: `answer`, `sources_used`, `confidence`, `needs_human`.

That’s an example, not a template. The sections are the point, and what goes under them changes with the job.

Some agents need one more thing in that first version: an escape hatch. If there are cases where the right move is to stop and hand off to a human, write that in from the start. Give it the trade-off in both directions, what handing off costs and what pressing on with a wrong answer costs, and phrase it as something the agent is allowed to do rather than a line it must never cross. Permission it can act on beats a prohibition it has to interpret. The escape hatch needs its own eval cases: one where the agent should hand off, and one where it shouldn’t. The first tells you whether it knows when it’s stuck rather than inventing an answer, which is the failure you actually care about. The second catches the over-correction that follows.

## Escalation
Hand off to a human when the tools you have can't resolve the request, or when
the next action is irreversible.

Escalating costs a person's time and adds a day to the response. Not escalating
when you should costs a wrong answer that someone acts on. Weigh both sides.
If you can't tell which side you're on, escalate.

Always set `needs_human` (true/false). When it's true, fill `handoff_summary`
with what you tried, what you found, and the decision the human needs to make.

Add a line only when a real failure earned it

After the first version, the only thing that should grow the system prompt is a failure you actually saw. The agent fails, the failure becomes a case in the eval set, and then you decide how to handle it. Every line in a mature system prompt should trace back to a specific failure.

How you handle it matters more than the fact you handled it. The instinct is to turn the volume up: NEVER answer without checking. CRITICAL: always validate first. One becomes three, three becomes a ban list. Reach for structure instead, in one of three shapes:

  • A tool, when the agent couldn’t do the thing. Telling it to do the thing properly won’t help if you haven’t given it the means.
  • A field, when the output is the wrong shape or a step got skipped. Pin the schema, and make the step a required field it has to fill in.
  • A criterion, when it made the wrong judgement call. Usually that’s because you gave it one side of a trade-off. State both sides and let it reason.

The same set of failures, written both ways. What not to do:

## Rules
NEVER answer a pricing question without checking the price list.
CRITICAL: always get the discount maths right.
NEVER answer without citing a source.
NEVER escalate unless absolutely necessary.

What to do instead:

## Tools
`lookup_price(sku)` - for every pricing question. Don't answer from memory.
`apply_discount(order, code)` - for any discount maths.

## Output
JSON: `answer`, `sources_used` (required, non-empty), `needs_human`.

## Escalation
Escalating costs a person's time. Not escalating a case that needed it costs a
wrong answer someone acts on. If you can't tell which side you're on, escalate.

All three turn a wish into something the system can check. You can grade a field. You can’t grade a “NEVER”.

The order of your output schema is not cosmetic

The last one comes straight back to where this started, and it’s the most direct consequence of the model being autoregressive. Whether you’re describing a JSON shape in the system prompt or enforcing a Pydantic model on the request, the model fills that schema in the order you wrote it, one token at a time. Every token is conditioned on the ones before it, so every field it finishes becomes input to the fields that come after.

So the order is a design decision. Put decision first and assessment second and you get a decision generated with nothing behind it, followed by an explanation written to fit a call that was already made. Put the assessment first and the decision is generated with that assessment sitting in the context it’s reading from. It’s also the order a person would work in. Nobody decides and then assesses, you assess and then you decide, and the schema should read the same way round. Fields you put early become the working the later fields are computed on, which is the chain of thought mechanism showing up somewhere most people don’t look for it.

# The decision is made on nothing, then justified.
class Review(BaseModel):
    decision: Literal["approve", "reject"]
    assessment: str

# The decision is made on the assessment.
class Review(BaseModel):
    assessment: str
    decision: Literal["approve", "reject"]
    confidence: float

The rule I use: order the fields the way you’d want a person to work the problem. Evidence, then assessment, then the call, then anything about the call like confidence or a handoff flag. Conclusions go last. It costs nothing to get right and nothing to fix later, which makes it the cheapest thing on this list.

Worth reading

- Ben