The rl/ directory contains the Daydream reinforcement learning (RL) project. The project is separate from the main Daydream package. There is no daydream rl command. A Uniform Resource Locator (URL) identifies a network or repository location.

The project has two parts:

Path	Purpose
`rl/daydream_review_v1/`	A verifiers environment for review, fix, and test rollouts.
`rl/train/`	A prime-rl Group Relative Policy Optimization (GRPO) configuration.

The environment is a separate uv project. It pins verifiers==0.2.1. This separation keeps the verifiers dependency tree out of the main Daydream lock file.

Rollout task

One task represents one harvested pull request (PR). The task uses a repository image for the recorded PR head commit. The task also stores the recorded base commit and repository test command.

One rollout performs these actions inside the container:

The harness starts Daydream in non-interactive mode.
The harness accepts the fix gate with --yes.
Daydream runs the default deep review flow.
Daydream applies accepted fixes.
Daydream runs the repository tests.
Daydream commits successful fixes to the in-container repository.
The environment reads the archived run artifacts.
The environment runs the repository tests again when tracked code changed.

All model calls pass through the verifiers interception server. Parallel Daydream calls remain separate branches in the captured trace. The harness rejects a rollout when the trace contains no captured model calls.

The rollout container does not receive repository credentials. The image already contains the repository snapshot. The repository remote points to an in-container mirror. Any Daydream push stays inside the container.

Build sandbox images

The image system uses two layers:

images/base.Dockerfile installs Daydream and the agent command-line programs.
images/repo.Dockerfile adds one repository snapshot and its dependencies.

The base image pins Python 3.12.13 slim by an immutable image digest. It also pins these tool versions:

Tool	Version
uv	`0.11.29`
Claude Code	`2.1.214`
Codex	`0.145.0`
Node.js	`22.17.1`
Pi	`0.82.1`

The Claude layer checks the release-key fingerprint and the signed release manifest. It then checks the downloaded executable against the manifest checksum. SHA-256 is a cryptographic file-checksum algorithm. The Codex and Node.js layers check downloaded archives against pinned SHA-256 values before extraction. The default build supports amd64 and arm64 for these command-line programs.

The base Dockerfile provides INSTALL_CLAUDE, INSTALL_CODEX, and INSTALL_PI build arguments. Each argument defaults to 1. Set an argument to 0 in a direct Docker build to omit that backend program. The rollout harness rejects an image when the selected backend program is absent.

Add each repository to images/manifest.toml. Each entry requires a clone URL, image name, and test command. The setup_cmds list is optional and defaults to an empty list. The image name must not include a tag.

The RL image pipeline reads a corpus of imported pull requests. See Benchmarking for the import commands.

Build the base image and all PR snapshot images:

cd rl/daydream_review_v1
uv sync
uv run python images/build_images.py --corpus /tmp/daydream-corpus-train

Build images for one repository:

uv run python images/build_images.py \
  --corpus /tmp/daydream-corpus-train \
  --only acme/widgets

Use --no-base to reuse daydream-rl/base:latest. Use --base-only to build only the base image. Use --red to select the included fixture with an intentional test failure. This option checks that the builder rejects a failing baseline. Use --manifest PATH to select an image manifest. The default path is images/manifest.toml.

Each repository image tag is the first 12 characters of the PR head commit. The final image layer runs the repository test command. A failed baseline test makes the image build fail. The builder continues with other PR images and returns exit status 1 after any image failure.

Load the task set

The task set reads the imported benchmark corpus. It also reads images/manifest.toml.

The loader applies these checks:

It rejects repositories in the Daydream training exclusion list.
It rejects a task that has no base commit.
It rejects a repository that has no image manifest entry.
It warns when an indexed PR has no review snapshot commit.
It loads reference comments for metrics.

The exclusion list contains the held-out benchmark repositories. The same check applies to training and evaluation task sets. There is no option to bypass the check.

Training and evaluation use different corpus directories. The task set does not split one corpus through a command option.

The test fixtures under rl/daydream_review_v1/tests/fixtures/ support tests and examples. The pallets/itsdangerous manifest entry is a reference input. These files are not a released training corpus.

Evaluation backends

The environment can evaluate all three Daydream backends. Each backend uses one wire format:

Daydream backend	Evaluation wire format	Endpoint configuration
`claude`	Anthropic Messages	`ANTHROPIC_BASE_URL` and `ANTHROPIC_API_KEY`
`codex`	OpenAI Responses	A generated Codex provider configuration and `CODEX_INTERCEPT_KEY`
`pi`	OpenAI-compatible Chat Completions	A generated Pi provider extension and `VF_INTERCEPT_API_KEY`

The interception server supplies the endpoint and temporary secret. The harness writes required client configuration inside the container. The claude backend is the default evaluation backend.

Run the included Docker evaluation configuration from the environment directory:

uv run eval @ configs/eval-docker.toml \
  -m POLICY_MODEL_ID \
  --client.base-url https://policy.example.test/v1

The included evaluation files set push = false. Keep the /v1 suffix on the client base URL.

configs/eval-stub.toml points to the local fixture and stub service. Start both inputs before you run the evaluation:

uv run python -m daydream_review_v1.fixture /tmp/daydream-rl-smoke/repo
mkdir -p /tmp/daydream-rl-smoke/archive /tmp/daydream-rl-smoke/home
uv run python -m daydream_review_v1.stub_upstream --port 8399 &
uv run eval @ configs/eval-stub.toml

The stub run checks capture, artifact, and reward wiring. The stub replies do not measure review quality.

Reward axes

The task has two reward axes:

Axis	Source
`intrinsic_composite`	The Daydream trajectory scorer reads archived review artifacts.
`fix_tests_pass`	The environment runs the repository test command after a tracked fix.

The intrinsic calculation is deterministic for fixed artifacts. fix_tests_pass repeats the configured test command against the fixed tree. A nondeterministic repository test can change the test reward.

The intrinsic score uses the same scoring function as corpus annotation. It uses available correctness, grounding, format, and length signals. A run with no findings has no grounding value. The reward boundary maps that missing composite to 0.0. The w_composite task setting multiplies this score. Its default value is 1.0.

fix_tests_pass is 1.0 when the final test command succeeds after a fix. The value is 0.0 when the command fails. The w_tests task setting multiplies this value. Its default value is 1.0. A rollout with no tracked fix uses no_fix_reward. The default no_fix_reward value is 0.0. Repository images must pass their baseline tests before a rollout. Therefore, this axis measures the change made during the rollout.

golden_overlap records the share of reference comments whose file also has a Daydream finding. It is not part of the reward. The trace records the reward breakdown and the available axes.

Do not disable Daydream archive or evaluation output for these rollouts. The reward reader requires those artifacts.

Training backend limit

Evaluation accepts Anthropic Messages, OpenAI Responses, and Chat Completions. Training accepts only the Pi Chat Completions path at the pinned revisions.

The training renderer sends token identifiers and log probabilities through Chat Completions. The pinned training client does not implement the other two formats. Set the training harness backend to pi.

The standalone environment pins verifiers 0.2.1. The prime-rl v0.7.0 workspace contains verifiers fork commit 6c64ce6. Run the environment test suite inside the prime-rl workspace before a training run. This test detects an incompatible verifiers interface.

GRPO configuration

rl/train/rl.toml contains the prime-rl v0.7.0 configuration. GRPO is the default prime-rl algorithm at this revision.

The current configuration contains these values:

Setting	Value
Training steps	`500`
Sequence length	`32768`
Checkpoint interval	`50`
Training precision	`bfloat16`
Low-rank adaptation rank and alpha	`16` and `32`
Optimizer learning rate	`1e-5`
Batch size and group size	`128` and `16`
Maximum completion tokens	`8192`
Training workers	`8`
Daydream fanout per rollout	`4`

The configuration uses Docker for training and evaluation rollouts. The training and evaluation sections use separate corpus paths. Both paths in the repository file are placeholders. The image manifest path is also a placeholder.

The model value in the repository file only satisfies configuration validation. Select the policy model when you start prime-rl. The repository does not identify that placeholder as a trained model.

Run the configuration check from a prime-rl v0.7.0 workspace:

uv run rl @ /opt/daydream/rl/train/rl.toml \
  --dry-run \
  --model.name POLICY_MODEL_ID

Install rl/daydream_review_v1 into that workspace before the check. The training README contains the exact workspace and verifier revision procedure.

The Daydream repository does not include a released training corpus. The repository also does not contain a model produced by this configuration.

Back to Daydream