Browse documentation

Milestones

Report how far a long task got by checking named steps against a Turn's events, and say when the events cannot settle it.

View Markdown

A long task can stop half way. "Failed" says that it did. A milestone says where. A milestone is a step the agent should reach on the way, written as a question about the events of the Turn: did the platform start the sandbox, did the agent call a tool, did it write the file.

Run it

cd examples/cookbooks
pnpm milestones --agent agt_01J... --version 3

See Evaluation cookbooks for the key, the address of your stack and the flags every recipe takes.

Two sources, kept apart

Question	Source	Durable?
How did the Turn end?	The Turn (`runs.turns`)	Yes, for as long as the Run is retained
What happened on the way?	The Turn's events (`runs.events`)	No. A rolling window that expires event by event

The outcome is always the Turn's. Events are evidence about the way there, and evidence is only as good as the history it came from. Every page of events carries history.status, and the recipe reads it from the first page:

`history.status`	What the recipe does with a milestone the events do not show
`complete`	Reports it as `missed`. The events begin at the Turn's first, so it did not happen
`partial`	Reports it as `unknown`. Older events expired, and it may have been among them
`expired`, `empty`	Reports it as `unknown`
unavailable	Reports it as `unknown`. The API could not read the events, which is not the same as there being none

A milestone the events do show is reached in every case: a retained event proves it.

The exit code follows the same rule. It is 1 when a Turn did not complete or a milestone was missed, and 3 when nothing failed but a milestone was unknown. That is a run that could not be judged, and a gate must not read it as a pass.

How it works

A milestone is an id, a description and a question about the events:

export const MILESTONES: readonly Milestone[] = [
  {
    id: 'sandbox-ready',
    description: 'the platform started the sandbox',
    reached: (events) => sawPlatformEvent(events, 'sandbox.ready'),
  },
  {
    id: 'used-a-tool',
    description: 'the agent called a tool',
    reached: (events) => toolCalls(events).length > 0,
  },
  {
    id: 'wrote-report',
    description: 'the agent wrote report.md',
    reached: (events) =>
      toolCalls(events).some(
        (call) =>
          call.name === 'write' &&
          JSON.stringify(call.input ?? '').includes('report.md'),
      ),
  },
];

Platform events (source of salambo, such as sandbox.ready) and the agent's tool events (source of pi, such as tool_execution_start with a toolName) come from runs.events.list(runId, { turn_id }).

wrote-report counts a write only when the call also finished without an error: it pairs the tool_execution_start with the tool_execution_end that has the same toolCallId and reports isError as false. A start proves only that the agent tried. A write that failed while the Turn still completed and said it was done is exactly the case a milestone exists to catch. Payloads in events are bounded, so match on short, stable fields such as a tool name or a file name. Consumers must accept event types and fields they do not know, so a milestone looks for what it needs and ignores the rest.

Read the output

write-report: Turn failed, answer errored, events complete (5)
milestone      status   what it means
-------------  -------  --------------------------------
sandbox-ready  reached  the platform started the sandbox
used-a-tool    reached  the agent called a tool
wrote-report   missed   the agent wrote report.md
Furthest milestone: used-a-tool. Not reached: wrote-report.

The header line gives the Turn's status, whether the answer met the case (unchecked when the case has nothing to check it against), and how complete the events were. The exit code is 0 when every milestone was reached, 1 when a Turn did not complete or a milestone was missed, and 3 when a milestone was unknown.

Change it

Edit MILESTONES in src/02-milestones.ts, in the order the agent should reach them, and pass your own task with --cases. Good milestones are steps that show progress and would be visible even if the task then failed: a tool that fetched the input, a file that was written, a deployment that was started.

A milestone is not the outcome. An agent may reach every milestone and still answer wrongly, and may skip one and still be right. Use task completion for the outcome.

PreviousTask completion

Up nextPolicy checks