daydream benchmark runs the private PR benchmark. The benchmark imports review cases from one GitHub repository, builds a Harbor dataset, and scores Daydream findings against golden reviews. A pull request (PR) is one review unit.

The legacy daydream bench verb and its withmartian offline corpus are removed. daydream bench prints an error and returns exit status 2.

The implementation is in daydream/benchmark/. The detailed repository runbook is docs/benchmark.md.

Command surface

Subcommand	Purpose
`init`	Creates a private benchmark workspace.
`status`	Shows read-only derived workspace state.
`validate`	Validates the workspace. Exit codes are 0, 2, and 1.
`import-prs`	Imports selected PRs as review cases.
`curate`	Curates the golden review for one case.
`build-harbor`	Packages the workspace for Harbor.
`upgrade`	Repairs legacy case documents.
`run`	Runs the supervised Harbor review.
`calibrate-judge`	Runs an optional judge diagnostic.
`clean`	Removes derived or all workspace artifacts.
`objective`	Writes one completed run as JSON.
`aggregate`	Pools a suite manifest into one report.

Each subcommand takes the workspace directory first:

daydream benchmark init ~/bench-owner-repo --repo OWNER/REPO \
  --reviewer-host openrouter.ai --judge-host openrouter.ai
daydream benchmark import-prs ~/bench-owner-repo --pr 123
daydream benchmark build-harbor ~/bench-owner-repo --daydream-wheel dist/daydream.whl
daydream benchmark run ~/bench-owner-repo --yes

Prerequisites and privacy boundary

Install these programs before you run the benchmark:

Harbor 0.22 or later but before 0.23. Install it in the same Python environment as Daydream.
Docker Desktop for macOS in its standard local configuration.

OrbStack, proxy network bridges, Linux Docker hosts, and public-network fallbacks are not supported. The first build and run performs a live check against the local engine.

Pi is the reviewer backend. OpenRouter is the provider for the reviewer and the judge. The base container runs with no network access. Only an explicit host allowlist opens egress. Set the allowlist at init with --reviewer-host and --judge-host. A run refuses a review or judge edge with no allowlisted host.

Gold reviews stay hidden from the reviewer. A run passes an Oracle self-match gate before Daydream trusts its findings. Credentials stay in the launching shell environment. The commands never write credentials into the workspace.

Import, curate, and run

Import private evidence explicitly. Each imported case is frozen into a deterministic source snapshot before anything reviews it:

daydream benchmark import-prs ~/bench-owner-repo --pr 123
daydream benchmark import-prs ~/bench-owner-repo --pr-file prs.txt
daydream benchmark curate ~/bench-owner-repo --case CASE_ID --apply-gold gold.yaml
daydream benchmark validate ~/bench-owner-repo
daydream benchmark run ~/bench-owner-repo --oracle --yes

run starts a supervised Harbor job in a no-network container. --oracle requires the run to pass the Oracle reward gate before the findings count toward trust. The reviewer model and judge route come from the compiled Harbor profile.

Read objective data

objective reads one completed ledgered run. It writes strict JSON with only opaque identifiers, counts, precision, recall, F1, token totals, and cost. It emits no repository names, PR numbers, source text, or judge reasoning:

daydream benchmark objective ~/bench-owner-repo --run-id RUN_ID --json -

aggregate reads a suite manifest and pools several runs into one report.

daydream benchmark aggregate suite.json --json -

Both commands are read-only. They do not start Harbor and do not call a judge.

Score meaning

The judge compares each candidate finding with the curated golden review. Daydream sums true positives, false positives, and false negatives across cases. Daydream then calculates aggregate precision, recall, and F1 from those sums.

Clean up

daydream benchmark clean ~/bench-owner-repo --derived
daydream benchmark clean ~/bench-owner-repo --all --yes

Back to Daydream