osprey

What it is

Osprey is an agentic coding orchestrator in Rust, built around a directed-search self-improvement loop. The agent runs as a persona-driven, multi-turn loop with streaming LLM drivers and an interactive terminal UI (TUI). Its system prompt and tool descriptions are an editable surface that a harness searches against a frozen, stratified slice of real terminal tasks under a token-penalized reward, accepting only variants whose gains clear a measured noise margin.

The unit of evaluation is a full multi-turn, tool-using agent episode on a real filesystem, averaging 48 agent steps and 554k tokens per task. Every candidate evaluation is expensive and noisy. The harness's center of gravity is statistical decision machinery for that setting.

What it's for

Osprey exists to answer a research question: can a coding agent measurably improve its own behavior by searching over the text that defines it? You can also run it as a plain coding agent through its TUI, without engaging the search loop at all.

Self-improvement loop

A directed-search harness runs the loop. Each iteration mutates the agent's editable surface, re-evaluates it against the frozen 20-task slice, and accepts or rejects the change via git tags. Published measurements so far are the baseline and the noise floor: no candidate has yet cleared the gate.

Editable surface

Per-iteration mutations are restricted to two override files, materialized by the harness and pointed at via environment variables:

• The default persona file (core/osprey-agent/personas/default.md): the base persona instructions that form layer 1 of the composed system prompt. Overridden at runtime via OSPREY_SYSTEM_PROMPT_FILE.
• Per-tool descriptions (core/osprey-tools/prompts/tool-descriptions.toml): the text surfaced to the model for each of the eight core tools. Overridden via OSPREY_TOOL_DESCRIPTIONS_FILE.

The agent loop, persona resolver, drivers, and tool dispatcher are deliberately out of scope: they are the substrate, not the editable parameters. Expanding the surface to include scaffold knobs, persona contents, and per-tool argument schemas is on the roadmap.

Reward

score = mean(reward) - lambda * mean(tokens)

• reward is per-task pass or fail in [0, 1], scored by each benchmark task's own test harness.

• tokens is total tokens consumed by the agent during the task.

• lambda is derived, not hand-tuned. It is computed at runtime from the recorded baseline:

  lambda = 0.05 / (0.10 * mean_tokens)
  

  With the current baseline (mean_tokens = 553,942), this gives lambda approximately 9.03e-7. The formula is designed so that a 10% token bloat costs roughly 5% of the reward budget on the slice. It is tied to the slice version: changing the slice forces a fresh baseline run and a re-derived lambda.

The token penalty is what keeps the search honest. Without it, the optimizer is free to buy marginal pass-rate gains with unbounded verbosity.

Slice

The slice freezes 20 benchmark tasks, stratified from a baseline run into five groups: 7 easy passes, 5 medium passes, 1 hard pass, 5 small fails, and 2 big fails (slice version v1-2026-04-08). The mix matters. A pure pass-only slice has no headroom to improve against. A pure fail-only slice has no floor to protect.

Accept gate

Five full re-runs of the unchanged agent established the noise floor. Per-run score standard deviation was 0.089 (a 12.8% coefficient of variation), with individual tasks flipping pass/fail up to 40% of the time.

A single comparison against that backdrop would accept noise as progress. Each iteration instead runs a paired trial: five evaluations of the incumbent champion and five of the candidate. Acceptance requires the paired delta to exceed 1.5 times the pooled standard deviation, a margin calibrated to roughly a 0.9% false-accept rate against the measured floor. A positive-but-sub-noise delta is rejected.

Search

The harness runs a directed search on a single linear branch:

1. Propose: a pluggable backend writes candidate override files into a staging directory. The default backend invokes a headless coding agent; any backend matching the protocol can be substituted (a scripted sweep, a separate agent, or a human in the loop).
2. Apply: the harness validates the edit allowlist (only the two files above) and scrubs for API-key-shaped literals before writing.
3. Evaluate: run the paired champion and candidate trials through the benchmark adapter, 10 evaluations total per iteration.
4. Score: compute the token-penalized reward.
5. Accept: pin an optimize/accepted/<N> git tag to HEAD.
6. Reject: reset hard back to the previous accept tag.

Guardrails bound the loop. It refuses to operate on main or any non-optimization branch. A file lock enforces one iteration at a time. A baseline fingerprint check halts the run if the agent's config, candidate inputs, or slice version diverge from the recorded baseline. Host provider keys are stripped from the proposal subprocess's environment: the proposer CLI keeps only its own key. One documented gap remains: the max-turns override sits outside the fingerprint's scope.

Baseline

The baseline run (model: deepseek/deepseek-v4-flash): 14 of 20 tasks passing (70%), a mean of 553,942 tokens and 47.9 agent steps per task. A reward-hacking audit of the baseline, with explicit checks for test-file modification, reward-file writes, and solution-directory access, came back clean across all five audited trials.

Relation to prior work

Self-improvement at the paper level is a crowded field. Osprey's position in it is a deliberate set of trades. DSPy's GEPA and MIPROv2 optimize prompts for LM pipelines where rollouts are cheap; Osprey pays for end-to-end agent episodes and spends its rigor on the accept statistics. The Darwin Godel Machine and SICA let the agent rewrite its own code, maximizing search-space breadth at the cost of auditability; Osprey confines mutation to two text files behind a hard allowlist, so any change the gate accepts lands as a readable diff pinned by a git tag. SICA's cost-penalized utility is the closest prior art to the token-penalized reward. What Osprey adds over all of these is the production-harness setting: a measured noise floor and a calibrated paired accept gate for an objective where a single evaluation is a wall-clock-capped half-hour of real terminal work.

Substrate

The execution infrastructure the loop sits on top of:

• Persona-driven multi-turn agent loop. Bundled default, architect, and verifier personas. Resolution searches project, user, and bundled tiers. The architect and verifier personas are read-only, each with a terminal tool (submit_plan, submit_verification) and a JSON-schema argument shape.
• Eight core tools: read_file, write_file, edit_file, bash, find_files, grep, list_directory, and tree, exposed through a gRPC sandbox sidecar. A conditional ninth tool, retrieve, is advertised when context compression is enabled, letting the agent fetch byte-exact originals of compressed tool results.
• Subagent orchestration. Spawn, send-input, wait, and cancel control tools with an eight-child concurrency cap shared across the entire spawn tree. Persona allowlists gate which tools children can access. Nested spawning is bounded by persona configuration. Children persist as forkable sessions and are viewable live in the TUI.
• Skills. The agent discovers skills from .agents/skills/ under the project root, ~/.agents/skills/ globally, and any extra roots passed via --skill-dir <PATH> (or the OSPREY_SKILL_DIR environment variable, which is repeatable). Each root is scanned for subdirectories containing a SKILL.md with agentskills.io frontmatter. Skill names follow the agentskills.io spec: lowercase alphanumeric plus hyphens, 1-64 characters, no leading, trailing, or consecutive hyphens. Namespaced names of the form namespace:slug (for example beagle-rust:review-rust) are accepted; the slug segment must match the parent directory. Discovered skills are surfaced to the model in an <available_skills> block and loaded on demand via the load_skill tool. The --skill <name> flag on osprey agent forces a named skill's SKILL.md body into the system prompt at session start as a <forced_skill> block, independent of model-driven load_skill. Multiple --skill flags are supported. An unknown skill fails fast at startup, listing available skills. Project-local skills shadow extra-dir skills of the same name, which in turn shadow user-global ones.
• Bring-your-own-model. Direct Anthropic, OpenAI, and OpenRouter drivers with Server-Sent Events (SSE) streaming and per-role driver and model selection.
• ATIF v1.6 trajectory logging. Standardized execution traces with agent schemas, per-turn timing, and tool-call records. This is the input format the reinforcement learning roadmap consumes.
• Local-first session persistence. SQLite by default; Postgres opt-in for shared installs. Sessions survive crashes and can fork from any past turn.
• Sandbox. macOS isolation via sandbox-exec with a deny-by-default network policy (loopback only) and a per-repo allowlist for outbound hosts. Other platforms run unsandboxed.

Roadmap

Two phases scope the next round of self-improvement work. Framework choice is deferred until each phase starts so the field can be re-evaluated.

Composable evaluation and multi-environment optimization. Replace the single scalar reward with a weighted combination of dimensions: task success, token efficiency, and RLAIF (Reinforcement Learning from AI Feedback) judge scores for soft quality. Replace the single benchmark with an environment trait so any eval harness can serve as one of many objectives. Surface per-turn ATIF trajectory data so the harness can identify which turn in a multi-step run failed, not just whether the task passed.

RL training pipeline and model self-improvement. Convert ATIF trajectories into scored trajectory groups, export them as datasets for GRPO (Group Relative Policy Optimization), DPO (Direct Preference Optimization), or supervised fine-tuning (SFT), and wire the environment into a trainer that updates model weights. A model hot-swap to a local inference server closes the loop: execute, score, train, deploy, repeat. A distillation pipeline (large-model trajectories into a smaller agent model) and reward-hacking detection are explicit scope items. The committed piece is that strategic intent: a model fine-tuned for Osprey's own harness. The specific integration shape is deferred.

Status

Osprey is under active development. It is distributed publicly through the osprey-public repo as pre-built binaries with an install script, so you can run the tool today. The main source repo remains closed during active development. The optimization loop is operational: the baseline and noise floor are measured and the automated outer loop is wired. Improvement numbers will be published as candidates clear the accept gate. This page tracks the methodology and architecture as the work progresses.

Install

Install the latest release with the one-line script:

curl -fsSL https://raw.githubusercontent.com/existential-birds/osprey-public/main/install.sh | bash

Then configure a provider and launch the TUI:

osprey setup
osprey

Pre-built binaries for each release are on the releases page.

Supported platforms

You need an API key for one of Anthropic, OpenAI, or OpenRouter; SQLite is built in, so there is no separate database to set up. macOS runs sandboxed via sandbox-exec; Linux runs unsandboxed.