Ussama Assad Start a project

Domain 4 of 5 · 20% of the exam

Prompt Engineering & Structured Output

What it covers

Explicit criteria, few-shot examples, structured output through tool use and JSON schema, validation and retry, batch processing, multi-pass review. About 20% of scored items, by the official exam guide.

Know cold

One idea per card. Open the ones you aren't sure of.

Explicit criteria

Say what to report and what to skip, with an example of each. "Flag anything important" is not a criterion.

Pick the technique for a reason

Start zero-shot. Add a few examples when the format or judgement is easier to show than describe, and include an edge case. Ask for step-by-step reasoning when the path to the answer matters. On some newer models Anthropic's docs say to rely on the model's built-in thinking rather than asking it to write its reasoning out.

Structure through tool use

A tool with a JSON schema gives you structured output. Make fields nullable and say "null if not stated", or a required field will get filled with something invented.

Types are not relationships

Schema validation checks each field. A rule across fields, such as a total that must equal the sum of its lines, needs a model-level validator (in Pydantic, model_validator).

Retry with the error

A retry that resends the same prompt is hoping. Send the failed output and the specific validation error. Only retry what can be fixed; if the information isn't in the source, stop and route it to a person.

Message Batches API

Half the standard price, most batches done within an hour, up to 24 hours, no latency guarantee. Poll until processing_status goes from in_progress to ended, then read results and match them by custom_id. Resubmit only the failed ones. Results stay available for 29 days. Batch is for work nobody is waiting on.

Review

A model reviewing its own output in the same session is anchored to its own reasoning. Use a fresh instance with a narrow job it can check against the source. Tier it: cheap checks on everything, deeper checks on a sample, people on the outliers.

Streaming

Each content block arrives as content_block_start, then deltas, then content_block_stop. The start event carries the block type, which is how a client routes text and tool calls.

Traps

  • "It passed the schema, so it's right". Schema validation checks fields one at a time. A rule between fields needs a model-level validator.
  • Retrying the same prompt. Send the failed output and the exact error. If the information isn't in the source, don't retry: return null and route it to a person.
  • A required field with no way out. If the model can't say "not present", it fills the field with something plausible. Make it nullable.
  • Batching work someone is waiting for. Batches have no latency guarantee. Use them for work nobody is waiting on.
  • Self-review in the same session. The model is anchored to its own reasoning. Use a fresh instance with a narrow job.

Practice: 4 questions

Q13 · Domain 4 · Prompt Engineering & Structured Output

You extract the delivery date from scanned delivery notes. The instruction "Return the delivery date as YYYY-MM-DD" works on clean notes and fails on unusual layouts, where the date sits in a stamp or a handwritten box. What do you try next?

  1. A. Fine-tune a model on a few thousand labelled delivery notes before going any further.
  2. B. Ask for step-by-step written reasoning on every single note before it gives its final answer.
  3. C. Add a few examples of awkward layouts with the correct output, including one with no date (null).
  4. D. Raise max_tokens so that the model has more room to search each note carefully.
Answer and explanation

Answer: C. The desired behaviour is easier to show than to describe, which is what few-shot examples are for, and including a no-date case stops the model inventing one. Fine-tuning is a last resort. Step-by-step reasoning helps when the reasoning path matters; spotting a date in a stamp is recognition. max_tokens caps output length and has nothing to do with this.

  • A. Fine-tuning is a last resort. Showing the model a few examples comes first.
  • B. Step-by-step reasoning helps when the reasoning path matters. Spotting a date in a stamp is recognition.
  • C. Right. The behaviour is easier to show than to describe, which is what few-shot examples are for. The no-date example stops the model inventing a date.
  • D. max_tokens caps output length. It has nothing to do with where the date sits on the page.

Q14 · Domain 4 · Prompt Engineering & Structured Output

Timesheet extraction returns daily_hours (a list of numbers) and week_total. Every output passes schema validation, yet some week_total values don't equal the sum of daily_hours. What is the fix?

  1. A. Add minimum and maximum bounds to week_total in the schema so that out-of-range totals get rejected.
  2. B. Add a model-level check that sum(daily_hours) equals week_total, and retry with that exact error.
  3. C. Set the temperature to zero so that the arithmetic comes out identical on every single run.
  4. D. Append the sentence "Always double-check your totals before answering" to the prompt.
Answer and explanation

Answer: B. Schema validation checks each field's type on its own. A rule that relates two fields needs a model-level validator (in Pydantic, a model_validator). Feeding the failed output and the specific error into the retry gives the model something to correct. Bounds check one field in isolation, temperature zero can be consistently wrong, and a reminder in the prompt is advice.

  • A. Bounds check one field on its own. A total can be in range and still not match the days.
  • B. Right. Schema validation checks each field on its own. A rule that relates two fields needs a model-level validator, and sending the specific error with the retry gives the model something to correct.
  • C. Temperature zero makes the output consistent, not correct. It can be consistently wrong.
  • D. A reminder in the prompt is advice. Nothing checks that it worked.

Q15 · Domain 4 · Prompt Engineering & Structured Output

Your extraction schema marks purchase_order_number as a required string. Many documents don't have one, and reviewers keep finding plausible PO numbers that appear nowhere in the source. What change fixes this at the cause?

  1. A. Add a regex pattern to the field so only well-formed PO numbers pass validation.
  2. B. Tell the model in the system prompt that it must never make anything up.
  3. C. Retry the extraction until the model returns a number matching the pattern.
  4. D. Make the field nullable and say "return null if the document doesn't state it".
Answer and explanation

Answer: D. A required field with no way to say "not present" pushes the model to fill it. A regex lets a well-formed invented number straight through, the prompt line is advice, and retrying until it matches rewards the invention. When the information really isn't in the source, don't retry at all: return null and route to review if it matters.

  • A. A regex lets a well-formed invented number straight through.
  • B. The prompt line is advice, and the required field still pushes the model to fill it.
  • C. Retrying until it matches rewards the invention. When the information isn't in the source, don't retry at all.
  • D. Right. A required field with no way to say "not present" pushes the model to fill it. With null allowed, a missing PO number comes back as null, and you can route that to review if it matters.

Q16 · Domain 4 · Prompt Engineering & Structured Output

Each night you classify about 40,000 support transcripts for a dashboard people read in the morning. During the day a chat widget classifies each message while the user waits. How should the Message Batches API be used?

  1. A. Batch the nightly job, poll until processing_status is "ended", then read results. Keep the widget live.
  2. B. Send both workloads through the Batches API, because batching halves the cost of every request you make.
  3. C. Put the widget traffic and the nightly job into the same batch so that both of them share one queue.
  4. D. Batch the nightly job and read all of its results straight after the create call returns a response.
Answer and explanation

Answer: A. Batches cost 50% of standard prices and most finish within an hour, but processing can take up to 24 hours with no latency guarantee. That suits work nobody is waiting on and fails a live user. The create call returns at once with processing_status "in_progress"; results are ready when it reads "ended", and custom_id ties each result to its request.

  • A. Right. Batches suit work nobody is waiting on. Poll until processing_status reads "ended", read the results, and match them to requests by custom_id. The widget stays on the normal API.
  • B. Batches have no latency guarantee and can take up to 24 hours, which fails a user waiting in the chat widget.
  • C. Widget traffic in a batch waits as long as the batch does. A live user needs the answer now.
  • D. The create call returns at once with processing_status "in_progress". Results are ready only when it reads "ended".