Task completion

Did the agent complete the task? This recipe answers with two separate facts, both read from the Turn, the durable record of what was asked and answered:

1. Did the Turn complete? That is the platform's word: completed, or failed, incomplete, cancelled.
2. Does the answer do what the case asks? That is a check you write.

A case passes when both hold. A Turn that did not complete is reported as errored, apart from a completed Turn with a wrong answer, because they need different fixes: the first sends you to the Run's failure, the second to the agent's behavior.

Run it

cd examples/cookbooks
pnpm task-completion --agent agt_01J... --version 3

See Evaluation cookbooks for the key, the address of your stack and the flags every recipe takes.

How it works

For each case the recipe starts a Run of the exact version, waits for the Turn, then reads the Turn back from its own history:

const run = await salambo.runs.execute({
  agent,
  version,
  input: evalCase.input,
  background: true,
});

// Wait for that Turn to reach a status that never changes. The deadline is the
// recipe's own: a wait has none, and aborting it does not cancel the Turn.
await salambo.runs.wait(run.id, {
  turn_id: run.turn.id,
  signal: AbortSignal.timeout(timeoutMs),
});

// The durable record, and the version it was pinned to.
const record = await salambo.runs.turns.retrieve(run.id, run.turn.id);

The recipe then checks record.agent_version against the version it asked for. If they differ it stops and scores nothing, because the result would be about another version.

runs.wait reads the Turn with growing gaps between reads and outlasts a rate limit. If a Turn has not finished when --timeout-minutes passes, the wait is aborted, and the recipe cancels the Turn with runs.cancel, because aborting a wait does not, and reports it as timed_out. Ctrl-C does the same for every Turn in flight. See Wait for a turn.

Read the output

case        turn       answer   took   detail
----------  ---------  -------  -----  ------------------------------------------------------------
arithmetic  completed  passed   4.2s   391
json-shape  failed     errored  900ms  provider provider_unavailable: The model provider is unavai…
summary     completed  wrong    6.3s   missing "sandbox"

1/3 passed, 1 wrong answer, 1 Turn that did not complete.

The exit code is 0 when every case passed and 1 otherwise, so the recipe works as a gate in CI.

Change it

The check is mustInclude and mustNotInclude on each case, matched against the final answer without regard to letter case. Pass your own cases with --cases:

[
  {
    "id": "refund-window",
    "input": "Can I return an order after 45 days?",
    "mustInclude": ["30 days"]
  }
]

A case with neither mustInclude nor mustNotInclude is refused, because a Turn that completed says nothing about whether its answer is right. Pass --completion-only to measure only whether Turns complete. It applies to every case: answers are not graded, even for a case that has checks, so a pass means the Turn completed and no more. The recipe says so.

For a check a string cannot make, use a policy check on what the agent did, or the LLM judge on what it said.

Why the Turn and not the events

The answer and the outcome are read from runs.turns, not from events. Events are a rolling window: older ones expire while newer ones remain, so an evaluation that depended on them would start returning different results as a Run aged. The Turn stays readable for as long as the Run is retained. See Read turn history.