osprey

Benchmark and optimization

Source-checkout profiles run evaluations, reports, dashboards, and prompt searches that can rewrite Git history.

Scope

Benchmark and optimization code is not part of the installed osprey command. These procedures require access to the source checkout at the audited commit.

The repository defines profiles for Terminal-Bench 2.1 and SlopCodeBench. Each profile pins its dataset digest, scoring rules, task settings, and run settings.

Benchmark runs call model providers and can create large local artifacts. The default jobs directory is the sibling path ../osprey-bench/jobs. A full run can use 2 gigabytes (GB) to 12 GB.

Prepare the checkout

The next commands install developer tools and Harbor 0.20.0. They register the checkout adapter. They store Harbor authentication. They also download benchmark images.

OSPREY_REPOSITORY_ROOT="$HOME/code/osprey"

cd "$OSPREY_REPOSITORY_ROOT" || exit 1

make install-tools

make setup-harbor

harbor auth login

make bench-images

make install-tools installs Cargo utilities and the Linux musl target. make setup-harbor installs Python packages into the active Python environment. Use an isolated Python environment when the active environment is shared.

Run benchmarks

make bench-smoke runs one task with one concurrent worker. make bench uses the selected profile. The profile supplies the trial and concurrency counts.

cd "$OSPREY_REPOSITORY_ROOT" || exit 1

make bench-smoke

make bench

Set BENCH=slopcodebench to select the SlopCodeBench profile. Terminal-Bench 2.1 is the default profile.

make bench-live starts a model run and a live local dashboard. The target requires MODEL and defaults PROVIDER to openrouter.

cd "$OSPREY_REPOSITORY_ROOT" || exit 1

MODEL_NAME="z-ai/glm-5.2"

make bench-live MODEL="$MODEL_NAME" PROVIDER="openrouter"

The dashboard binds to loopback by default. Its --lan mode binds without authentication. Use --lan only when every host on the network is authorized to read benchmark data.

Validate and report

Validation reads a completed run without modifying its files. Report generation writes a separate output tree.

Set RUN_DIRECTORY to an existing run. The report command creates or updates REPORT_DIRECTORY. Replace run-id in both variables before you run these commands.

RUN_DIRECTORY="$HOME/code/osprey-bench/jobs/run-id"

REPORT_DIRECTORY="$HOME/code/osprey-reports/run-id"

cd "$OSPREY_REPOSITORY_ROOT" || exit 1

make bench-validate RUN="$RUN_DIRECTORY"

make report RUN="$RUN_DIRECTORY" REPORT_DIR="$REPORT_DIRECTORY"

Optimization surface

The optimizer allows each candidate to change only these two files:

  • core/osprey-agent/personas/default.md
  • core/osprey-tools/prompts/tool-descriptions.toml

Each run uses a 20-task Terminal-Bench 2.1 slice. The default limit is 150 turns and 1,800 seconds for each evaluation. Each gate compares five champion trials with five candidate trials. The configured noise multiplier is 1.5.

The score is mean(reward) - lambda * mean(tokens). The harness derives lambda from the recorded baseline mean token count.

Current measurement records

The committed baseline uses slice v2-2026-06-28 and model z-ai/glm-5.2. The record reports 17 passes from 20 tasks, 203,146.5 mean tokens, and 40.7 mean steps. Its status is out_of_band because the pass rate is 0.85.

The committed noise record uses slice v1-2026-04-08. Do not combine that noise record with the current v2 baseline. The committed tag set contains only optimize/accepted/0. That tag points to 74e7bb2 on a separate history. The repository therefore contains no accepted candidate measurement for this target.

Preflight the destructive loop

The optimizer commits candidates and creates acceptance tags. It resets rejected candidates with git reset --hard. Run it only on a dedicated optimize/ branch with a clean worktree. Preserve unrelated work outside the checkout before this command.

The audited checkout fails the accepted-tag ancestry check. Do not run the optimizer against this tag state. The following preflight stops when the baseline tag is not an ancestor or the worktree is not clean.

cd "$OSPREY_REPOSITORY_ROOT" || exit 1

OPTIMIZE_BASE_TAG="optimize/accepted/0"



if ! git merge-base --is-ancestor "$OPTIMIZE_BASE_TAG" HEAD; then

  printf '%s\n' "The optimization baseline tag is not an ancestor of HEAD." >&2

  exit 1

fi



if test -n "$(git status --porcelain --untracked-files=all)"; then

  printf '%s\n' "The optimization worktree is not clean." >&2

  exit 1

fi

The source maintainer must reconcile the accepted tag before an optimization run on this target. After reconciliation, create a dedicated optimize/ branch before make optimize. The accept code rejects main and other branch prefixes. A file lock permits one optimization iteration at a time.

The optimizer writes untracked ledgers and trial matrices under bench/optimize/. Those files survive a hard reset. The harness also writes benchmark jobs outside the repository by default.

Guardrails and failures

The optimizer compares three fingerprints with the baseline record. The fingerprints cover live configuration, candidate input, and the slice. A mismatch stops the run and requires a new baseline record. Runtime max-turn settings are outside that fingerprint.

The proposer subprocess receives only its required credentials. The apply step rejects paths outside the allowlist. It scans for provider-key-shaped text.

Docker rate limits can interrupt image downloads. Run make bench-images before a long evaluation.

Ledger and tag disagreement indicates an interrupted acceptance transition. Inspect bench/optimize/results.tsv and optimize/accepted/* before another run.

The optimizer utility also requires the Harbor package. Without make setup-harbor, the Python entry point fails during import. The proposer requires an installed and authenticated Claude Code command that supports claude -p. Before reconciling the accepted tag, run bench/optimize/scripts/probe-environment.sh. The script checks the Claude Code flags -p, --output-format, --allowed-tools, and --add-dir. It also checks harbor run --task, a reachable Docker daemon, Git, and the platform lock dependency. The lock dependency is flock on Linux or Python 3 with fcntl on other supported systems.

A nonzero Claude Code exit adds a propose_edit_subprocess_error row to the optimization ledger. A proposal that changes neither allowlisted file adds a no_op row and creates no candidate commit. The proposer timeout is 600 seconds. A missing claude executable or an expired timeout is not handled by the nonzero-exit branch. Either condition exits the outer Python command without adding that subprocess-error row.

Back to Osprey