# Milestones

Report how far a long task got by checking named steps against a Turn's events, and say when the events cannot settle it.

A long task can stop half way. "Failed" says that it did. A milestone says where. A milestone is a step the agent should reach on the way, written as a question about the events of the Turn: did the platform start the sandbox, did the agent call a tool, did it write the file.

## Run it

```bash
cd examples/cookbooks
pnpm milestones --agent agt_01J... --version 3
```

See [Evaluation cookbooks](/docs/cookbooks/overview) for the key, the address of your stack and the flags every recipe takes.

## Two sources, kept apart

| Question                  | Source                            | Durable?                                         |
| ------------------------- | --------------------------------- | ------------------------------------------------ |
| How did the Turn end?     | The Turn (`runs.turns`)           | Yes, for as long as the Run is retained          |
| What happened on the way? | The Turn's events (`runs.events`) | No. A rolling window that expires event by event |

The outcome is always the Turn's. Events are evidence about the way there, and evidence is only as good as the history it came from. Every page of events carries `history.status`, and the recipe reads it from the first page:

| `history.status`   | What the recipe does with a milestone the events do not show                                          |
| ------------------ | ----------------------------------------------------------------------------------------------------- |
| `complete`         | Reports it as `missed`. The events begin at the Turn's first, so it did not happen                    |
| `partial`          | Reports it as `unknown`. Older events expired, and it may have been among them                        |
| `expired`, `empty` | Reports it as `unknown`                                                                               |
| unavailable        | Reports it as `unknown`. The API could not read the events, which is not the same as there being none |

A milestone the events do show is `reached` in every case: a retained event proves it.

The exit code follows the same rule. It is 1 when a Turn did not complete or a milestone was `missed`, and 3 when nothing failed but a milestone was `unknown`. That is a run that could not be judged, and a gate must not read it as a pass.

## How it works

A milestone is an `id`, a description and a question about the events:

```ts
export const MILESTONES: readonly Milestone[] = [
  {
    id: 'sandbox-ready',
    description: 'the platform started the sandbox',
    reached: (events) => sawPlatformEvent(events, 'sandbox.ready'),
  },
  {
    id: 'used-a-tool',
    description: 'the agent called a tool',
    reached: (events) => toolCalls(events).length > 0,
  },
  {
    id: 'wrote-report',
    description: 'the agent wrote report.md',
    reached: (events) =>
      toolCalls(events).some(
        (call) =>
          call.name === 'write' &&
          JSON.stringify(call.input ?? '').includes('report.md'),
      ),
  },
];
```

Platform events (`source` of `salambo`, such as `sandbox.ready`) and the agent's tool events (`source` of `pi`, such as `tool_execution_start` with a `toolName`) come from `runs.events.list(runId, { turn_id })`.

`wrote-report` counts a write only when the call also **finished without an error**: it pairs the `tool_execution_start` with the `tool_execution_end` that has the same `toolCallId` and reports `isError` as false. A start proves only that the agent tried. A write that failed while the Turn still completed and said it was done is exactly the case a milestone exists to catch. Payloads in events are bounded, so match on short, stable fields such as a tool name or a file name. Consumers must accept event types and fields they do not know, so a milestone looks for what it needs and ignores the rest.

## Read the output

```text
write-report: Turn failed, answer errored, events complete (5)
milestone      status   what it means
-------------  -------  --------------------------------
sandbox-ready  reached  the platform started the sandbox
used-a-tool    reached  the agent called a tool
wrote-report   missed   the agent wrote report.md
Furthest milestone: used-a-tool. Not reached: wrote-report.
```

The header line gives the Turn's status, whether the answer met the case (`unchecked` when the case has nothing to check it against), and how complete the events were. The exit code is 0 when every milestone was reached, 1 when a Turn did not complete or a milestone was `missed`, and 3 when a milestone was `unknown`.

## Change it

Edit `MILESTONES` in `src/02-milestones.ts`, in the order the agent should reach them, and pass your own task with `--cases`. Good milestones are steps that show progress and would be visible even if the task then failed: a tool that fetched the input, a file that was written, a deployment that was started.

A milestone is not the outcome. An agent may reach every milestone and still answer wrongly, and may skip one and still be right. Use [task completion](/docs/cookbooks/task-completion) for the outcome.