The End of Harness Engineering

Abstract. A harness is the program written around a model, and writing it by hand has been the skill. The paper argues that this is ending. A hand-written harness is one guess about one task on one afternoon; once it ships, it runs for months, unread and unscored. In the loop the paper describes, the harness is a file that grades itself: a small 4B model reads what failed and proposes 8 edits, and only the edit that scores best on held-out questions survives. This repeats every round. The big model stays frozen; only the file around it changes.

Index terms: harness engineering, self-improving harness, held-out evaluation, frozen model, loop engineering

1 Introduction

Harness engineering has meant writing the program around the model by hand. A person studies one task, writes one file, and ships it. After that nobody reads the file and nobody scores it, so a wrong guess keeps running for months. The paper's answer is to stop writing that file by hand and let a loop write it, with the model itself untouched.

2 The loop

Each round starts from what failed. A small 4B model reads the failures and proposes 8 edits to the harness file. Every edit is scored on held-out questions, the best one is kept as the next version, and the other seven are dropped. The big model is never retrained: it stays frozen while the file around it changes.

Fig. 1. Two ways to write a harness. The hand-written path is a single guess; the loop-written path reads failures, proposes edits, and keeps only what survives held-out scoring.

Algorithm 1One round of the loop

1 Input: harness h_0, frozen model M, scoring set Q

2 F := failures of M running h_n on the task set

3 E := 8 edits proposed by the 4B editor from F

4 for each e in E: s(e) := score of e on Q

5 h_{n+1} := h_n + the e with the highest s(e)

6 n := n + 1; repeat

3 What changes

Three parties touch the system and only one of them is a person. The human writes the scoring set: the questions the harness is graded on. The 4B model writes every edit to the harness file. The big model writes nothing; it is the same frozen model in every round, and only the file around it changes.

Table 1 Who writes what in the loop

Party	Writes	How often
Human	the scoring set	once, before the loop
Loop, 4B model	harness edits	8 per round
Big model	nothing, frozen	runs every round

Fig. 2. Held-out score by round. Each round the editor proposes eight edits; the seven rejected are grey, the kept one is orange. The harness changes only when an edit wins.

4 Conclusion

When a loop writes the harness, frameworks become disposable and the exam becomes the job. The order of work flips: first the questions the harness will be graded on, then the code. That inversion is the paper's real claim, and it is a governance claim as much as an engineering one, because the scoring set is the artifact a human still owns.

One guess about one task on one afternoon.
4B: a model of about four billion parameters, small next to the frozen model it edits for.
Held-out questions are kept out of the editing step and used only for scoring.