Wordsmith

Whole-book editorial benchmark · V0.1

A book begins as fragments. Fred sees where they converge.

Fred builds a whole-book view from dispersed manuscript evidence. This benchmark tests whether that view produces more useful editorial decisions than a sophisticated Claude Opus 5 workflow.

80%Fred

20%Claude Opus 5

Highest-priority whole-book findings the writer would act onActionable = adopt or modify after a blind review.
Thirty-six manuscript fragments resolve into nine grouped representations of people, places and concepts, then into three higher-order editorial views of throughline, themes and tensions.PEOPLEPLACESCONCEPTSTHROUGHLINETHEMESTENSIONSFRED · WHOLE-BOOK STATEMANUSCRIPT FRAGMENTSMANUSCRIPT PROGRESSIONACCUMULATING WHOLE-BOOK CONTEXTThirty-six fragments resolve into nine middle-layer clusters, then three higher-order editorial views.

By the Wordsmith team.

Published . Explanation updated .

This is a product-team study. Read the method and limitations alongside the results.

Results

What the benchmark found

The study keeps judgment, breadth, and deliverable editorial yield separate. It does not collapse performance into a single score.

Priority judgment

80% Fred · 20% Claude Opus 5

The share of each system's highest-priority whole-book findings the writer would adopt or modify after a blind review.

Editorial breadth

772 Fred · 201 Claude Opus 5

Editorial contributions returned across the manuscript. This measures how much work came back ready to inspect, not whether every contribution was correct.

Editorial yield

5.4× more professionally deliverable suggestions

Fred returned 161 professionally deliverable suggestions, versus 30 from Claude Opus 5. This measures the usable editorial work returned, rather than rewarding a shorter list for being easier to keep clean.

AI-detector result

99% Human before · 99% Human after

On a human-written book-length manuscript, accepting 73 executable Fred edits left Pangram 4.0's whole-book classification unchanged. A complete Claude Opus 5 rewrite of a frozen passage registered Mixed, confirming detector sensitivity.

Method

What was tested and how to read the results

Fred and Claude Opus 5 received the same whole-book editorial assignment. Their returns were reviewed without system labels. Judgment, breadth, and professionally deliverable output were scored separately.

  • Manuscript A: 36,978 words. Later, radically shorter manuscript. Controlled capability, precision, and complete-return case.
  • Manuscript B: 91,092 words. Earlier, substantially longer predecessor. Complete-return and author-action case.

The study separates controlled capability checks, precision review, blind author decisions, and complete-return delivery audits. Each result applies to its own review unit, not to all books or all suggestions.

Result provenance

  • C1-CAPABILITY: Controlled capability. Matched defects, negative controls, and the locked 30-check profile.
  • C1-PRECISION: Precision review. The clean one-shot suggestion comparison, including the equal 34-item budget.
  • C1-AUTHOR: First blind author review. The preserved 25-decision sample for each system.
  • C1-DELIVERY: First complete return. Deterministic contribution and placement audit for the first return.
  • C2-AUTHOR-AUDIT: Second blind author review. The 23 change-recommendation sensitivity after the protect instrument failed its pre-reveal audit.
  • C2-DELIVERY: Complete-return delivery. Deterministic placement and normalized anchor audits for the complete returns.

These identifiers describe the internal review records supporting the reported aggregates. They are not links to a public dataset. No private manuscript text or individual reviewer record is published on this page.

Scope

Limitations

  • This is a Wordsmith-produced study, not an independent certification. Wordsmith has a commercial interest in the result.
  • The sample contains two related manuscript versions: a 36,978-word later manuscript and its 91,092-word predecessor. It is not a representative sample of writers, genres, or books.
  • The comparison concerns the declared Claude Opus 5 workflow and the tested Fred configuration. It does not establish superiority over every Claude prompt, model, editor, or future product version.
  • Priority judgment uses the ten highest-priority findings from each system. Breadth counts returned contributions; it does not certify each contribution as correct. Precision-review yield and complete-return breadth have different denominators and must not be combined into one score.
  • Blind author decisions reflect this writer's judgment. They are not a substitute for an independent expert panel, publication outcomes, or a guarantee that another writer will adopt the same share of suggestions.
  • The Pangram result is one detector observation on one manuscript and one set of accepted edits. It does not prove authorship, unchanged literary voice, or acceptance by publishers, competitions, or other detectors.
  • Only aggregate results and this method summary are published here. The underlying manuscripts and individual reviewer records are not publicly downloadable, so this page is not a fully reproducible open dataset.

What this means for your manuscript

A benchmark is context, not a verdict on your book. Judge the read on evidence from your own manuscript and decide which changes belong in it.

WordsmithWhole-book editorial benchmark · V0.1
The Wordsmith Whole-Book Editorial Benchmark, V0.1 | Wordsmith