Autonomous harness — how it works, how to drive it

1The workflow: three phases, one fresh window each

Fresh-window-per-phase is the context-control layer. Everything a later window needs travels through GitHub labels, the issue body, and the local ledger — never through a chat transcript.

2How to prompt it — worked examples

Each command takes plain language; the harness derives titles, slugs, and labels. These are copy-paste starting points.

Filing (window 1) — with and without an explicit type

The bracketed kind: is optional. Explicit wins; otherwise /new-issue infers it (GPU comparison → experiment, variation of a PASSed parent → ablation, code-only → implementation, GPU-free offline study → analysis, open-ended → brainstorm). The kind decides the lifecycle shape — analysis runs no GPU; implementation/brainstorm/literature close on the plan/PR itself — so state it whenever inference could guess wrong.

/build "does signed_ema α=0.4 hold parity at cadence 10/10?"
# kind inferred → experiment
/build "sweep anchor β over {0.0,0.25,0.5} on the PASSed #63 setup" kind:ablation
/build "add a code-execution reward adapter for DeepCoder (sandboxed tests)" kind:implementation
/build "offline: is consecutive-update cosine predictive of projector gain?" kind:analysis
/build "reading list: RLVR-linearity papers relevant to one-layer training" kind:literature

Planning (window 2) — spend ALL uncertainty here

/plan 64          # fast tier: single hypothesis, one launch round
/plan 64 deep     # multi-stage; add constraints as plain lines, e.g.:
Stage 1 is a 2-step probe on the cheapest rung; gate stage 2 on val@25 ≥ 0.70.
Budget cap 24 GPU-hr. If Big-Math OOMs, shrink MAX_RESPONSE_LENGTH before
escalating the ladder.

Deep tier is the sanctioned place for judge panels, clarifying questions, and codex-verify. After approval there are no review loops anywhere.

Executing (window 3) — pick how the GPU appears

/execute 64                 # gpu_mode from the plan / project default (auto)
/execute 64 --gpu auto      # hands-off: provisions itself once CPU gates are green
/execute 64 --gpu ask       # prepare everything, then PAUSE "READY FOR GPU"
# …come back later with the box you rented (login registered automatically):
/execute 64 --attach 43811234
# …or change your mind and let it provision:
/execute 64 --gpu auto
# account + attach at the stage level too:
/launch 64 --attach 43811234 --account team

Operator-managed side box (never auto-reaped)

bash .claude/skills/vast-attach/run.sh --exp-id 64-ema-sweep --issue 64 \
     --instance-id 43811234 --manual   # analysis/download box you own end-to-end

3The 7 stages underneath (never merged, never skipped)

The phase commands are a UX layer. The state machine is these stages — each a self-contained skill, each flipping its own label. /go N resumes from whatever labels + ledger say and runs what remains.

Overview any time: /status (prints to stdout — there is deliberately no status file to go stale).

4Where state lives (one local session file, nothing else)

5Plans live in the GitHub issue — nowhere else

The planner publishes the plan into the issue body between <!-- plan:start --> / <!-- plan:end -->. There are no local plan files — the only thing on disk is a gitignored fetched cache. Machine fields sit in the plan's first ```yaml fence, parsed by _lib.sh plan_field.

• Edit the plan on GitHub freely — your edits are authoritative; every gating stage (/plan, /approve, /launch) re-fetches before acting.
• Checkbox discipline: at close, every success-criteria / progress box the verdict proves is ticked in place (plan_tick — literal substring match, refuses to push a truncated body). A closed issue never shows unticked boxes; a box the verdict marks ✗ stays unticked and is named in the close comment.
• After launch the plan is snapshotted to runs/<id>/run.json and never read again — execution is decoupled from the plan.

6GPUs: hands-off vs ask-me (and bring-your-own-box)

Implementation always FINISHES on the laptop before any box exists (verification.gpu_idle_rule): the runner's PREPARE phase — branch, patch, launch payload, one bounded CPU sanity pass — completes before provisioning starts, in every mode. A GPU is never up while code is being authored; real on-policy numerics are only ever tested on the GPU.

Resolution order for the mode: CLI flag → plan gpu_mode: key → project.yaml default_compute.gpu_mode (auto). Every ssh/rsync offers the registered key (project.yaml vast_ssh) — bare ssh fails publickey.

7Two-tier watching: cheap eyes, expensive judgment

The default watcher is machine-monitor (Sonnet): purely mechanical 30-second health polls — ssh/tmux liveness, per-GPU util, global_step advance, disk, error-pattern grep, WandB heartbeat. Zero diagnosis by design. The Opus training-log-monitor is dispatched once per anomaly, in classify mode — you pay for judgment only when judgment is needed.

Hard rule: no adversarial verification, judge panels, or exploratory workflows during a run. Mid-run doubt = flag + stop for a human go/no-go.

8Results & publishing: the record leaves the laptop

/close runs a strict order, each step idempotent — so a crash anywhere resumes cleanly:

• 1 teardown check (money before paperwork), then SUMMARY row + launcher promotion + results PR → squash-merge onto the base branch.
• 2 tick the plan's checkboxes, then post the close comment (the verdict SSOT, with the deterministic report URL), then flip status:done.
• 3 publish (scripts/publish_run_report.py): one self-contained HTML page → Experiment Runs on this site, card inserted on the tab, small artifacts → the repo's gitignored artifacts/<id>/, bulk artifacts → R2 autonomous-harness-rlvr-compression/<id>/, then commit + push (the push IS the Cloudflare Pages deploy). A partial publish exits nonzero → needs:human, and cleanup is skipped.
• 4 cleanup sweep: run dir, plan cache, stale handles, ledger compaction, PROGRESS ticks — all gone. The checkout keeps only the SUMMARY row.

/de-bloat (human-only, DEBLOAT_OPERATOR_ACK=1) remains as the manual batch fallback for leftover dirs.

9Parallel issues = parallel windows in worktrees

claude --worktree 64-ema-sweep     # session A, issue 64
claude --worktree 65-cadence-20    # session B, issue 65 — zero cross-contamination

• State paths always resolve to the primary checkout (_lib.sh and the publish script both anchor via git worktree list), so all sessions share exactly one ledger (writes are lock-guarded).
• The autosave Stop-hook commits only that session's branch.
• One issue per window. Never two sessions driving the same issue.

10Pause signals & failure modes

Durable pause = a label on the issue. needs:human (with the reason as an issue comment) and awaiting:approval.

11Money: reaping, burn, budget

• /status or bash .claude/skills/vast-cost/run.sh — live $/hr on both accounts + untracked-box leak check.
• python scripts/check_budget.py --month — spend vs project.yaml budget: caps.
• EXTERNAL rows and boxes belonging to other users are never auto-destroyed.

12The hook suite (the harness's reflexes)

Hooks run without any agent deciding to run them. Config: .claude/settings.json; scripts: .claude/hooks/. Every external call inside them is timeout-bounded and none can block a session.

13Design doctrine (what must survive refactors)

• Labels + ledger first, files second. Deleted files ⇒ named refusal or terminal derivation, never a retry loop. Run dirs are evidence, not state.
• Every retry is a ledger counter (launch_attempts ≤ 3, fix_attempts ≤ 3, revise_depth ≤ iterations, monitor ≤ 12 cycles). Exhaustion ⇒ needs:human, never a spin.
• Verification doctrine: unbounded deliberation before approval; NONE autonomous during execution. Mid-run doubt ⇒ flag + stop.
• CPU-first, GPU-last: implementation finishes on the laptop; a box is provisioned last and never idles under code authoring; CPU checks are one bounded pass.
• Cheap eyes, expensive judgment: the mechanical watcher is the cheap model; the classifier is Opus, on demand. Every reasoning agent stays on the best model.
• Publish, then delete: the durable record is GitHub + this site + R2; the checkout keeps one SUMMARY row and one session file. Nothing else survives a close.
• One name everywhere: <N>-<slug> is the run dir, ledger id, Vast label, WandB group, and branch suffix; cells are self-describing kebab slugs.
• Self-hosting branch discipline: every issue gets exp/<id> from project.yaml source_tree.base_branch, pushed before launch; /close PRs results back to that same base on shamanez/verl — never upstream, never main. No branch name is hardcoded in a skill or agent.

AAppendix

Unattended multi-day run (optional — not the default path)

/bg /goal Issue 64 is terminal: status:done and box TORN_DOWN — or the issue
gained a needs:human/awaiting:approval label. Each turn: run /go 64 once,
print status + evidence. Stop after 120 turns.

Always include the label escape + a turn bound. The default workflow needs neither: one issue per window, human nearby.

Manual baseline launch (bypass the harness)

Provision with .claude/skills/vast-provision/run.sh (locked template verl-research-vllm020 — see project.yaml vast_template), ssh in, run examples/grpo_trainer/vast_baseline_qwen25_1p5b_grpo_gsm8k.sh.

One-time prerequisites

ls -l ~/.config/verl-research/secrets.env    # expect -rw------- (never echo values)
gh repo set-default --view                   # expect shamanez/verl-compression-research
which claude gh vastai uv jq aws