No single winner yet
Claude Fable 5.1 (high), Gemini 4 Argon (high), GPT-6.1 Sol (high), Claude Opus 5.5 (high), GPT-6 Astra (high) are within each other's margins of error.
Which AI edits books best?
Frontier AI models do the three jobs an editor does on a manuscript: craft notes, line editing and a final polish. The pages come from unpublished books, so no model has seen them. A panel of AI judges from different model makers grades the work.
Top overall
Claude Fable 5.1high
78.9 overall
Best craft notes
Claude Fable 5.1high
69.1 on craft
Best line editing
GPT-6.1 Solhigh
77.8 on line
Strong for the money
GPT-6.1 Solhigh
77.6 at $2.79 per 10k words
Overall score out of 100, with 95% intervals
Findings
Craft: Claude finds more, GPT is more careful
Claude Fable 5.1 (high) leads craft (69.1). GPT-6.1 Sol (medium) raises fewer notes, but almost all of them are real.
Line: GPT-6.1 Sol leads
GPT-6.1 Sol (high) scores 77.8 and rarely makes a harmful edit.
Effort matters as much as the model
Claude Opus 5.5 scores 76.9 at high effort and 62.3 at low.
Polish is near the ceiling
Every model catches almost all the errors; this track will get harder.
Method
- Craft
- Editorial notes on writing technique (show and tell, dialogue, point of view and more), from a checklist built by our editors. Each note must be real, useful and safe. Wrong notes cost more than missed ones.
- Line
- Sentence-level edits for clarity, rhythm and word choice that keep the writer's meaning and voice. Each edit must improve the page, and the edited page must beat the untouched one.
- Polish
- A final proofread: spelling, grammar, punctuation and wrong words, leaving style alone. Fixes must be right and needed. Frontier models are close to the ceiling here.
- Pages
- From unpublished fiction and nonfiction manuscripts, so no model has seen them.
- Judging
- A panel of AI judges from different model makers. No judge grades its own maker's models. Every note or edit must quote the page exactly.
- Scores
- 0 to 100. Being right counts more than doing more. Each score has a 95% interval.
- Held back
- The instructions, the checklist and the pages are not published, so models cannot be tuned to the test.
Limitations
- A Wordsmith-produced pilot, not an independent certification.
- A small sample: intervals are wide and will narrow as the bench grows.
- The AI judges have not yet been checked against human editors.
- Results describe the named model and effort setting on the date of the run.
For Fred against Claude on a whole book, see Wordsmith vs Claude.