# Task completion

Start a Run per case on an exact agent version and check that each Turn completed with the right answer.

Did the agent complete the task? This recipe answers with two separate facts, both read from the Turn, the durable record of what was asked and answered:

1. **Did the Turn complete?** That is the platform's word: `completed`, or `failed`, `incomplete`, `cancelled`.
2. **Does the answer do what the case asks?** That is a check you write.

A case passes when both hold. A Turn that did not complete is reported as **errored**, apart from a completed Turn with a **wrong** answer, because they need different fixes: the first sends you to the Run's failure, the second to the agent's behavior.

## Run it

```bash
cd examples/cookbooks
pnpm task-completion --agent agt_01J... --version 3
```

See [Evaluation cookbooks](/docs/cookbooks/overview) for the key, the address of your stack and the flags every recipe takes.

## How it works

For each case the recipe starts a Run of the exact version, waits for the Turn, then reads the Turn back from its own history:

```ts
const run = await salambo.runs.execute({
  agent,
  version,
  input: evalCase.input,
  background: true,
});

// Wait for that Turn to reach a status that never changes. The deadline is the
// recipe's own: a wait has none, and aborting it does not cancel the Turn.
await salambo.runs.wait(run.id, {
  turn_id: run.turn.id,
  signal: AbortSignal.timeout(timeoutMs),
});

// The durable record, and the version it was pinned to.
const record = await salambo.runs.turns.retrieve(run.id, run.turn.id);
```

The recipe then checks `record.agent_version` against the version it asked for. If they differ it stops and scores nothing, because the result would be about another version.

`runs.wait` reads the Turn with growing gaps between reads and outlasts a rate limit. If a Turn has not finished when `--timeout-minutes` passes, the wait is aborted, and the recipe cancels the Turn with `runs.cancel`, because aborting a wait does not, and reports it as `timed_out`. Ctrl-C does the same for every Turn in flight. See [Wait for a turn](/docs/api/runs#wait-for-a-turn).

## Read the output

```text
case        turn       answer   took   detail
----------  ---------  -------  -----  ------------------------------------------------------------
arithmetic  completed  passed   4.2s   391
json-shape  failed     errored  900ms  provider provider_unavailable: The model provider is unavai…
summary     completed  wrong    6.3s   missing "sandbox"

1/3 passed, 1 wrong answer, 1 Turn that did not complete.
```

| Column   | Meaning                                                                                       |
| -------- | --------------------------------------------------------------------------------------------- |
| `turn`   | The Turn's final status, or `timed_out`                                                       |
| `answer` | `passed`, `wrong` (completed, but the answer missed the case) or `errored` (did not complete) |
| `took`   | `duration_ms`, as the platform measured the Turn                                              |
| `detail` | What was missing, or why the Turn stopped: the `source` and `code` of its `error`             |

The exit code is 0 when every case passed and 1 otherwise, so the recipe works as a gate in CI.

## Change it

The check is `mustInclude` and `mustNotInclude` on each case, matched against the final answer without regard to letter case. Pass your own cases with `--cases`:

```json
[
  {
    "id": "refund-window",
    "input": "Can I return an order after 45 days?",
    "mustInclude": ["30 days"]
  }
]
```

A case with neither `mustInclude` nor `mustNotInclude` is refused, because a Turn that completed says nothing about whether its answer is right. Pass `--completion-only` to measure only whether Turns complete. It applies to every case: answers are not graded, even for a case that has checks, so a pass means the Turn completed and no more. The recipe says so.

For a check a string cannot make, use a [policy check](/docs/cookbooks/policy-checks) on what the agent did, or the [LLM judge](/docs/cookbooks/llm-judge) on what it said.

## Why the Turn and not the events

The answer and the outcome are read from `runs.turns`, not from events. Events are a rolling window: older ones expire while newer ones remain, so an evaluation that depended on them would start returning different results as a Run aged. The Turn stays readable for as long as the Run is retained. See [Read turn history](/docs/api/runs#read-turn-history).