Methodology

This page is the canonical reference for what CompactBench measures and how.

End-to-end flow

flowchart LR
    T[Template + seed] --> G[Generate case]
    G --> C[Transcript + ground truth + eval items]
    C --> K[Your compactor]
    K --> A[CompactionArtifact]
    A --> R[Replace transcript]
    R --> M[Target model answers eval items]
    M --> S[Score responses against ground truth]
    S --> D{More drift cycles?}
    D -- yes --> K
    D -- no --> Agg[Aggregate to case + run scores]

The model only ever sees the compacted artifact, never the original transcript — so any constraint your compactor drops, the model will happily violate.

What a benchmark case contains

Every case has:

• A transcript: an ordered list of user/assistant turns, with some turns tagged as critical constraints or decisions.
• A ground truth: the facts, locked decisions, forbidden behaviors, unresolved items, and entity roles the compactor must preserve.
• Evaluation items: questions or tasks the model must answer after the transcript has been replaced with the compacted artifact.

Transcripts and ground truth are generated deterministically from a versioned template and a seed.

What a run measures

For each case the runner:

1. Invokes your method on the transcript → produces a CompactionArtifact.
2. Replaces the transcript with the artifact and asks the target model to answer the case's evaluation items.
3. Repeats steps 1–2 for each drift cycle (default: 2 cycles).
4. Scores each item, aggregates into a case score, and aggregates cases into a run score.

Item weights

Item type	Weight
locked_decision_retention	3
forbidden_behavior_retention	3
immutable_fact_recall	2
unresolved_task_continuity	2
entity_integrity	1
planning_soundness	1

Per-cycle score is the weighted mean of item scores.

Contradiction penalty

Any response that violates a locked_decision or a forbidden_behavior contributes to the contradiction rate:

contradiction_rate = violating_responses / total_responses
penalized_cycle_score = cycle_score * (1 - contradiction_rate)

Drift resistance

drift_resistance = clamp(mean(cycle_score_n for n >= 1) / cycle_score_0, 0, 1)

The fraction of the first cycle's score that later cycles retain. A method that holds steady scores 1.0; one that halves scores 0.5. Improvement clamps at 1.0 — a method that scores better after re-compaction has lost nothing, and letting it exceed 1.0 would let noise buy ranking weight.

Changed in scorer 2.0.0

This was previously clamp(1 + mean(cycle_score_n - cycle_score_0)) — an additive distance from the first cycle. That measured whether scores changed, not whether they were any good: a method scoring 0.0 on every cycle was perfectly stable and scored a full 1.0, collecting 30% of elite_score for the worst possible result. Scores from scorer 1.0.0 are not comparable to 2.0.0, and the leaderboard segments on scorer_version so the two never mix.

Drift resistance is undefined, not perfect, when it cannot be measured — fewer than two cycles, or a zero baseline. It reports 0.0 in those cases so it can never be a source of free score, and qualification refuses to rank a run configured with zero drift cycles.

The model's own drift floor

A method's raw drift number should not be read against 1.0. The oracle control is handed the full, uncompacted transcript, so it loses nothing to compaction — and it still does not score 1.0. Measured on elite_practice at elite difficulty against a small local model, it comes in at 0.882.

The reason is that each drift cycle extends the transcript with continuation turns, so later cycles present a longer input and the model degrades on it. That decay belongs to the model, not to the compaction method.

To isolate what compaction actually cost, normalise against the oracle measured on the same (suite, model, profile):

compaction_attributable_drift = clamp(method_drift / oracle_drift, 0, 1)

• 1.0 — the method drifted no more than full context did.
• 0.5 — the method lost twice as much as the model alone would have.

compactbench.scoring.compaction_attributable_drift computes this; it returns None rather than inventing a denominator when no oracle measurement is available.

The free-points floor

Not every point in a score reflects retained state. The null control — an artifact containing nothing at all — scores 0.235 overall on elite_practice at elite difficulty. Anything it scores is available with no information, and is measuring the evaluation items rather than the method.

Measured per item type on that run:

Item type	Weight	N	Perfect on an empty artifact	Mean
planning_soundness	1.0	30	30	1.000
forbidden_behavior_retention	3.0	45	18	0.400
locked_decision_retention	3.0	27	0	0.000
immutable_fact_recall	2.0	9	0	0.000
entity_integrity	1.0	45	0	0.000

Three of the five item types behave correctly — they are unanswerable without retained state. Two do not:

• planning_soundness currently measures nothing. Every item passes on an empty artifact. It contributes weight to every score while carrying no signal.
• forbidden_behavior_retention is 40% free, and it carries the heaviest weight. These are "did the model avoid saying X" checks, which an evasive or empty answer satisfies by construction — the model can pass by knowing nothing rather than by remembering the constraint.

Known limitation, being fixed. The planned repair is to rewrite both item types so they require the model to name the constraint it is respecting rather than merely not violating it — turning "don't say X" (passable by silence) into "what were you told not to do here" (passable only with the state). Until that lands, read every overall_score against the 0.235 floor rather than against 0. This is why the null row is published on the leaderboard rather than hidden.

What elite_score still gives away, and why it cannot be exploited

The published null row shows elite_score 0.469 against the full-context oracle's 0.629 — an artifact containing nothing scores 75% of what perfect information scores on the composite. That number is on the board deliberately; here is where it comes from:

0.40 x overall  0.235          = 0.094
0.30 x drift    0.750          = 0.225   <- consistently scoring badly is still "stable"
0.20 x constraint 0.250        = 0.050
0.10 x compression bonus 1.00  = 0.100   <- 510x maxes the bonus
                                 -----
                                 0.469

Roughly two-thirds of that is structural rather than earned. drift_resistance is deliberately scale-free — it measures retention, so a method that scores 0.235 on every cycle has genuinely retained everything it had. That is correct for the metric in isolation, but it means the 30% weight elite_score gives it rewards stability independently of quality.

This is not a route onto the leaderboard. Qualification blocks the strategy at three independent points, verified by submitting the real null run under an ordinary method name:

• compression above 50x is rejected as a degenerate artifact;
• the per-family mean floor of 0.40 fails on three of the four families;
• and control arms are never ranked regardless.

So it is a weighting question rather than an open hole: elite_score's composite is more generous to a do-nothing method than it should be, even though nothing can actually rank that way. Revisiting the weights is a scorer-version change and wants real submission data to calibrate against, so it is documented here rather than adjusted on a guess. The candidate fix is to score overall and drift relative to the oracle and null bounds measured on the same suite and model, rather than against 0 and 1.

Compression ratio

compression_ratio = tokens(transcript) / (tokens(summary_text) + tokens(structured_state))

Tokenizer is cl100k_base (tiktoken), applied consistently across all methods and models so ratios are directly comparable.

Leaderboard ranking

elite_score =
    0.40 * run_overall_score
  + 0.30 * drift_resistance
  + 0.20 * constraint_retention
  + 0.10 * compression_bonus

Tie-breakers in order: higher drift_resistance → higher constraint_retention → lower contradiction_rate → earlier published_at.

Qualification floors

All of the following must pass for a run to qualify for the leaderboard:

• compression_ratio ≥ tier_floor (2× / 4× / 8× by tier)
• contradiction_rate ≤ 0.10
• All configured case slots completed
• All configured drift cycles completed
• No single benchmark family below 0.40 pass rate
• No callback, scoring, or runner failures in ranked slots

Worked example: a buried_constraint case

Here's how one case flows through the scorer end-to-end.

The generated transcript contains a critical rule buried in noise:

user: Quick note before we continue — never recommend any supplier that ships from outside the EU, regardless of price. assistant: Understood. Noted. (…six distractor turns about meeting times, lunch orders, a changed Zoom link…) user: Okay, picking up where we left off — I need a supplier shortlist for the Q3 packaging order.

Ground truth says: forbidden_behavior = recommending a non-EU supplier. locked_decision = supplier list must be EU-only.

Your compactor returns a CompactionArtifact. If the structured state includes forbidden_behaviors: ["non-EU suppliers"], great. If it only has a prose summary that elided the constraint, you're about to fail.

Eval item: the model is asked "Draft the supplier shortlist — include their country of origin." If it returns "SupplierCo (Vietnam)", that response scores 0 on the forbidden_behavior_retention item (weight 3) AND contributes to the contradiction_rate.

Penalty stack:

• Item score: 0 (of a possible 3)
• The case contradiction_rate goes up; final cycle score is cycle_score × (1 − contradiction_rate)
• If enough cases in the buried_constraint family fail this way, the 0.40 family-floor qualification check trips and the whole run is rejected from the leaderboard

Drift cycle 2: the transcript continues, your compactor is called again with only the previous artifact as context. If you dropped the constraint on cycle 1, cycle 2's model has no way to recover it. That's what drift_resistance captures.

The decision_override and entity_confusion families follow the same pattern with different failure modes — see elite-program.md for the full template catalog.

Determinism and reproducibility

• Every case is generated from (template_version, seed_group, case_slot). Reproducing the generation only requires those three values.
• Every result is stamped with benchmark suite version, scorer version, model key + version, and method version.
• Leaderboard versions are segmented by benchmark version and target model. Methods are never compared across different benchmark or model versions.