Corpus preparation

daydream corpus prepares archived Daydream runs for data analysis and model training. A corpus is a selected set of run records. JavaScript Object Notation (JSON) stores one data object. JSON Lines (JSONL) stores one JSON object on each line.

The command has three subcommands:

Command	Purpose
daydream corpus harvest	Add an outcome and a reward record to archived runs.
daydream corpus label	Add an authoritative human outcome to one run.
daydream corpus build	Convert selected archive records to JSONL.

daydream corpus harvest does not collect pull request comments for a benchmark. The legacy daydream bench harvest command is removed. Benchmark PR import is now daydream benchmark import-prs. See Benchmarking.

The implementation is in daydream/training/.

Annotate archived runs

Run the annotation pass:

daydream corpus harvest

The command uses the Daydream archive. The default archive is $DAYDREAM_ARCHIVE_DIR or ~/.daydream/archive. Use --archive-dir to select another archive. The GitHub application programming interface (API) supplies pull request data.

The annotation pass performs these actions for each indexed run:

1. It reads the archived manifest, findings, verifier results, and review output.
2. It resolves pull request information when GitHub data is available.
3. It derives an outcome label from pull request or local repository evidence.
4. It calculates the intrinsic reward from available review signals.
5. It appends one annotation to the archive index.

The intrinsic reward uses correctness, grounding, format, and review-length signals. Missing signals remain missing. A missing signal is not converted to a zero value. Maintainer outcome data is stored separately from the intrinsic reward.

Each annotation has an observation time. A pull request outcome also has a valid time. These two times let a later build select the information available at a specified time.

The archive write uses the evidence commit and reward version as a duplicate key. An unchanged annotation is counted as skipped when harvest processes that row again. A new reward version permits another annotation generation.

Harvest options

Option	Behavior
--session PREFIX	Processes session identifiers that start with PREFIX.
--archive-dir PATH	Selects the archive root.
--cache-dir PATH	Selects the GitHub response and progress cache. The default is ~/.daydream/harvest-cache.
--repo-clone-root PATH	Selects the repository clone cache. The default is CACHE_DIR/repos.
--gh-spacing-sec SEC	Sets the delay between rows. The default is 0.8.
--dry-run	Builds annotations without writing observations or progress markers.

This example uses a separate cache and a one-second GitHub delay:

daydream corpus harvest \
  --cache-dir .daydream/harvest-cache \
  --repo-clone-root .daydream/harvest-repositories \
  --gh-spacing-sec 1.0

The clone cache supplies repository history for evidence checks. Harvest fetches updates before it uses an existing cached clone. Harvest creates a blobless clone when the cached clone does not exist.

The progress cache lets a stopped run continue without another GitHub request for completed sessions. Use the same --cache-dir when you continue a stopped run. The progress log also skips completed sessions after a reward-version change. Use a cache without completion markers when you must process the completed sessions again.

--dry-run does not make the complete harvest operation read-only. Harvest can still update its API cache. Harvest can still create or fetch repository clones. Harvest can also add a missing base commit to an archived manifest.

One row failure increments errors and does not stop later rows. An exhausted GitHub rate limit stops the current sweep and sets aborted to 1. The command prints all counts in a summary object. Row errors and an aborted sweep do not change the command exit status. Check the printed errors and aborted values in automation.

Understand the reward record

The reward reducer uses these intrinsic values:

Value	Rule
Correctness	Maps consistent to 1.0, uncertain to 0.5, and contradicts to 0.0. The credit weight is 0.6.
Grounding	Uses the stored grounding rate from 0 through 1. The credit weight is 0.4.
Format	Sets the composite to 0.0 when structured review data is invalid.
Length	Counts characters in the review output. The bounded penalty weight is 0.2.

The length penalty starts after 2000 characters. The penalty reaches its maximum at 10000 characters.

Daydream calculates a weighted mean from the available credit values. Daydream adjusts the weights when one credit value is missing. Daydream then subtracts the bounded length penalty. The final intrinsic composite is between 0 and 1. The composite is absent when no credit value is available and the format is valid.

A mapped maintainer outcome adds comparison values to the reward record. The outcome mapping is accepted=0.0, contested=0.5, and rejected=1.0. Daydream compares that value with a reviewer outcome baseline. The baseline is the mean of at least ten prior outcomes. Daydream uses 0.5 before ten prior outcomes exist.

posterior_cost is the absolute distance between the mapped value and the baseline. This comparison does not change the intrinsic composite.

Add a human outcome

Use a full session identifier or a unique prefix:

daydream corpus label SESSION_PREFIX --outcome accepted

The allowed outcomes are:

• accepted
• contested
• rejected
• unknown

The unknown outcome records a human decision. An unknown outcome is different from a run with no label.

The prefix must match exactly one archived run. No match returns exit status 1. More than one match also returns exit status 1.

Use --archive-dir PATH to select another archive. The human outcome has precedence over an automated outcome.

Build a JSONL corpus

Build an accepted-only corpus:

daydream corpus build --out data/daydream.jsonl

The default build includes complete runs with an accepted maintainer outcome. The accepted outcome must contain pull request evidence. An added human label does not add a reward or pull request outcome evidence. Use --include-all-labels to include those manually labeled rows. Daydream always excludes repositories in the training exclusion list. Daydream also excludes repositories in the copyleft list by default.

The build reads only local archive data. It does not call GitHub or Git. It does not change archived manifests.

Build filters

Option	Behavior
--repo OWNER/REPO	Selects a repository. Repeat the option to select more repositories.
--label LABEL	Selects an outcome. Repeat the option to select more outcomes.
--status STATUS	Selects a run status. The default is complete.
--min-grounding VALUE	Sets an inclusive minimum grounding rate from 0 through 1.
--min-reward VALUE	Can include a run that has no accepted outcome label when its intrinsic reward meets the threshold.
--include-all-labels	Disables the accepted-only label filter. This option conflicts with --label.
--allow-copyleft OWNER/REPO	Permits one repository from the copyleft list. Repeat the option as necessary.

This example selects two repositories and one label:

daydream corpus build \
  --out data/accepted.jsonl \
  --repo acme/api \
  --repo acme/worker \
  --label accepted

--min-reward compares the stored intrinsic reward. The comparison does not subtract a maintainer outcome penalty.

Time selection

Use --as-of with a Coordinated Universal Time (UTC) ISO 8601 value:

daydream corpus build \
  --out data/snapshot.jsonl \
  --as-of 2026-08-01T00:00:00Z

The build selects the latest annotation observed at or before the specified time. The build ignores an outcome that became valid after the specified time. Intrinsic values recorded with the annotation remain available for reward-based selection.

The default selects the latest annotation. lineage.json records the resolved time for every non-empty build.

Stack limits

Use stack stratification to limit a dominant stack:

daydream corpus build \
  --out data/stratified.jsonl \
  --stratify-by stack \
  --max-stack-share 0.6

stack is the only supported stratification value. The default maximum stack share is 0.6. The value must be greater than 0 and not greater than 1.

Daydream calculates the stack cap from the input record count. The value is not a strict share of the final output. Records with an unknown stack form one group.

Build outputs

A normal build writes these files in the output directory:

File	Contents
daydream.jsonl	Training records. The name comes from --out.
schema.json	JSON Schema for one training record.
lineage.json	Included-session hash, annotation versions, time selection, and creation time.

Each JSONL record contains run identity, repository context, review output, outcome, grounding, and reward data. The record also contains references to the archived trajectory and diffs. The record does not embed the trajectory or diff file contents.

fix_diff_ref points to the reviewed input diff. The field keeps its schema version 1 name for compatibility. recommended_diff_ref points to the Daydream outcome diff when that file exists. Reason and action spans point to agent steps inside trajectory.json.

Daydream writes the JSONL file through a temporary file and an atomic replacement. Daydream also writes lineage.json with an atomic JSON write. Daydream copies schema.json after the JSONL replacement.

A zero-record build writes an empty JSONL file and schema.json. It removes an old lineage.json file from the output directory.

Use --dry-run to print record counts without writing files:

daydream corpus build --out data/check.jsonl --dry-run

Use --emit-schema-only to copy only schema.json:

daydream corpus build --out data/daydream.jsonl --emit-schema-only

Schema-only mode does not read the archive. Schema-only mode removes an old lineage.json file. Schema-only mode does not remove an existing JSONL file.

Back to Daydream