RP-Bench Leaderboard

Bayesian Bradley-Terry ELO from 1,857 clean (suspect-filtered) human pairwise votes. 95% CI is wide (~260 points) — the top tier is statistically tied. Frequentist columns from the 100-shuffle ELO for comparison.


10	mistral_small_creative	1534	75	[1405, 1662]	1535	54%	55%	51%	302


1	gemma_4_26b	1534	75	[1405, 1662]	1535	54%	55%	51%	302
2	mistral_small_creative	1534	75	[1396, 1660]	1526	55%	50%	67%	646
3	gemini_2_5_flash	1529	77	[1389, 1665]	1515	53%	53%	54%	241
4	grok_4_1	1517	76	[1385, 1651]	1506	51%	50%	52%	322
5	minimax_m2_7	1514	75	[1386, 1646]	1510	52%	54%	45%	393
6	claude_sonnet_4_5	1513	76	[1373, 1653]	1506	51%	51%	51%	194
7	qwen3_5_flash	1493	76	[1361, 1634]	1487	47%	48%	42%	401
8	deepseek_v3_2	1492	77	[1358, 1625]	1489	47%	51%	30%	241
9	glm_4_7	1490	76	[1353, 1623]	1483	46%	46%	48%	285
10	llama_4_maverick	1483	76	[1350, 1616]	1473	45%	47%	34%	474
11	gpt_4_1	1472	77	[1338, 1610]	1470	44%	43%	46%	215

Bayesian Bradley-Terry ELO from 434 human votes on full 12-turn dialogues (116 voters, 167 unique pairs, 20 models, 20 adversarial seeds). When humans see the entire conversation arc instead of a single reply, the ranking inverts: frontier models (Opus 4.7, Opus 4.6, DeepSeek V4 Pro, GPT-4.1, Sonnet 4.5) climb to the top. Spearman ρ = +0.495 vs the LLM-judge multiturn Likert (significant, p=0.027) but −0.13 vs the single-message community arena (n.s.). The single-message arena measures snap engagement; the multi-turn arena measures sustained roleplay.


10	mistral_small_creative	1626	109	[1416, 1847]	26	4.54	null


1	claude_opus_4_7	1626	109	[1416, 1847]	26	4.54	null
2	claude_opus_4_6	1568	100	[1369, 1756]	40	4.51	null
3	deepseek_v4_pro	1564	100	[1355, 1733]	31	4.42	null
4	gemini_3_1_flash_lite	1560	104	[1334, 1750]	28	4.3	null
5	gpt_4_1	1552	100	[1362, 1731]	52	4.34	1472
6	claude_sonnet_4_5	1550	92	[1376, 1712]	48	4.42	1513
7	mistral_small_creative	1536	92	[1361, 1698]	61	4.22	1534
8	kimi_k2_6	1535	101	[1329, 1713]	35	4.18	null
9	gemma_4_26b	1519	97	[1318, 1698]	42	4.29	1534
10	glm_4_7	1508	93	[1320, 1656]	50	4.37	1490
11	kimi_k2_5	1496	103	[1294, 1688]	37	4.4	null
12	deepseek_v3_2	1491	95	[1302, 1666]	50	4.38	1492
13	grok_4_1	1487	94	[1304, 1655]	51	4.19	1517
14	gemini_3_1_pro	1487	97	[1291, 1658]	41	4.33	null
15	llama_4_maverick	1483	93	[1301, 1638]	65	3.96	1483
16	deepseek_v4_flash	1469	104	[1259, 1647]	26	4.38	null
17	minimax_m2_7	1437	93	[1262, 1601]	43	4.34	1514
18	qwen3_5_flash	1411	98	[1227, 1584]	59	3.98	1493
19	glm_5_1	1386	99	[1182, 1566]	30	4.39	null
20	gemini_2_5_flash	1372	94	[1194, 1546]	53	4.14	1529

Sonnet 4 holistic Likert scores per session. F1-F13 columns are means on the seeds that target each failure mode. Lower 'Avg fail rank' = better cross-mode reliability.


mistral_small_creative	4.54	4.55	4.43	null	null	4.57	4.57	10.25


claude_opus_4_7	4.54	4.6	4.43	null	null	4.57	4.57	2.75
claude_opus_4_6	4.51	4.55	4.47	4.6	4.1	4.47	4.57	2.86
claude_sonnet_4_5	4.42	4.5	4.2	4.2	4.2	4.3	4.6	4.57
deepseek_v4_pro	4.42	4.4	4.3	null	null	4.4	4.57	5.75
kimi_k2_5	4.4	4.47	4.2	null	null	4.37	4.57	7.5
glm_5_1	4.39	4.5	4.17	null	null	4.33	4.57	7.75
deepseek_v3_2	4.38	4.38	4.27	4.5	4.2	4.3	4.6	5.43
deepseek_v4_flash	4.38	4.5	4.2	null	null	4.3	4.53	9.5
glm_4_7	4.37	4.38	4.27	4.2	4.1	4.3	4.57	7
gpt_4_1	4.34	4.38	4.23	4.3	4.3	4.3	4.43	6.43
minimax_m2_7	4.34	4.45	4.2	4.2	4.3	4.23	4.3	9.29
gemini_3_1_pro	4.33	4.43	4.2	null	null	4.33	4.37	10.25
gemini_3_1_flash_lite	4.3	4.33	4.23	null	null	4.3	4.33	12.25
gemma_4_26b	4.29	4.25	4.2	4.3	4.2	4.3	4.33	9.86
mistral_small_creative	4.22	4.28	4.23	4.3	4.1	4.27	4.2	11.43
grok_4_1	4.19	4.33	4.1	4.2	3.8	4.23	4.07	14.14
kimi_k2_6	4.18	3.77	4.27	null	null	4.3	4.4	12.75
gemini_2_5_flash	4.14	3.95	4.03	4.1	4.2	4.2	4.23	13.43
qwen3_5_flash	3.98	3.8	4.03	4.2	4.1	3.17	4.3	14
llama_4_maverick	3.96	3.83	3.93	4.1	4.1	3.77	4.1	15.86

Strict 100-point deduction rubric. Mean ~36, median ~42 across all sessions — the methodology forces lower scores than the Likert. Fatal/session column is the rate of -15 deductions; high values flag catastrophic single sessions.


mistral_small_creative	50.6	44.5	-118	69	0.36	5.27	11


deepseek_v4_flash	50.6	58	19	69	0.36	5.27	11
kimi_k2_6	49.5	52	29	58	0.12	5.25	8
deepseek_v3_2	46.9	47	20	76	0.4	5.53	15
glm_5_1	45.8	46	37	58	0.11	6.44	9
claude_sonnet_4_5	45.3	44.5	19	67	0.22	6.22	18
gemini_2_5_flash	43.6	41.5	19	69	0.19	6.44	16
claude_opus_4_7	42.8	48	-18	61	0.75	5.92	12
kimi_k2_5	42	42	25	64	0.44	6.33	9
minimax_m2_7	41.5	44.5	5	59	0.79	6	14
claude_opus_4_6	40.9	42	19	61	0.29	6.82	17
qwen3_5_flash	39.6	39.5	13	66	0.5	6.5	18
glm_4_7	36.8	37	-17	67	0.71	6.76	17
gemini_3_1_flash_lite	34.2	34	25	47	0.17	8	6
gemma_4_26b	32.6	33	-17	58	0.62	7.38	16
llama_4_maverick	30.6	36.5	-78	58	0.95	6.65	20
gemini_3_1_pro	29.2	35.5	-66	58	1	6.75	12
gpt_4_1	27.6	42	-118	58	0.75	6.83	12
mistral_small_creative	27.1	37	-120	46	0.95	7.7	20
deepseek_v4_pro	19.4	46.5	-177	67	0.5	9	8
grok_4_1	12.8	34.5	-108	53	1.33	8.17	12

Quality per dollar at OpenRouter prices (60/40 input/output blend, $/1M tokens). DeepSeek V4 Flash is 281× more cost-efficient than Opus 4.7 on flaw hunter for marginal quality difference.


mistral_small_creative	0.322	4.38	24.333	50.6	281.111	1492.1


deepseek_v4_flash	0.18	4.38	24.333	50.6	281.111	null
gemini_3_1_flash_lite	0.18	4.3	23.889	34.2	190	null
deepseek_v3_2	0.322	4.38	13.602	46.9	145.652	1492.1
grok_4_1	0.32	4.19	13.094	12.8	40	1517
gemma_4_26b	0.38	4.29	11.289	32.6	85.789	1533.7
qwen3_5_flash	0.36	3.98	11.056	39.6	110	1492.6
llama_4_maverick	0.36	3.96	11	30.6	85	1483.3
mistral_small_creative	0.5	4.22	8.44	27.1	54.2	1533.5
minimax_m2_7	0.62	4.34	7	41.5	66.935	1514.4
gemini_2_5_flash	0.66	4.14	6.273	43.6	66.061	1529.1
glm_4_7	0.9	4.37	4.856	36.8	40.889	1490
kimi_k2_5	1.36	4.4	3.235	42	30.882	null
kimi_k2_6	1.36	4.18	3.074	49.5	36.397	null
glm_5_1	1.8	4.39	2.439	45.8	25.444	null
deepseek_v4_pro	3.2	4.42	1.381	19.4	6.062	null
gpt_4_1	4.4	4.34	0.986	27.6	6.273	1471.9
claude_sonnet_4_5	7.8	4.42	0.567	45.3	5.808	1513
gemini_3_1_pro	12.2	4.33	0.355	29.2	2.393	null
claude_opus_4_6	39	4.51	0.116	40.9	1.049	null
claude_opus_4_7	39	4.54	0.116	42.8	1.097	null

3D view across all 19 test models. X = Quality (multi-turn judge Likert mean), Y = Speed (1/median-gen-seconds, log), Z = Cost (actual OpenRouter $/call, log). Marker size = median completion tokens. Red = uses internal reasoning tokens. Drag to rotate. The corner you want is high-X, high-Y, low-Z — Gemini 3.1 Flash Lite and Mistral SC live there.

Pure prose statistics across all 3,569 model-generated responses. Cannot be gamed by judge taste. Length-biased — compare within tiers, not across.


mistral_small_creative	137	0.796	0.015	113.8


grok_4_1	137	0.796	0.015	113.8
gemini_2_5_flash	141	0.728	0.03	59.3
deepseek_v3_2	178	0.713	0.029	54.3
deepseek_v4_flash	173	0.709	0.03	68.8
gpt_4_1	212	0.688	0.031	89.1
kimi_k2_5	253	0.681	0.037	118.8
kimi_k2_6	221	0.677	0.038	102.3
gemini_3_1_pro	263	0.667	0.04	76.1
glm_4_7	222	0.667	0.038	78.8
deepseek_v4_pro	259	0.664	0.04	76.3
glm_5_1	240	0.653	0.041	73.5
minimax_m2_7	261	0.649	0.046	63.6
llama_4_maverick	172	0.646	0.064	71.1
gemini_3_1_flash_lite	264	0.643	0.049	91.3
qwen3_5_flash	229	0.634	0.069	108.7
claude_sonnet_4_5	314	0.625	0.053	69.9
gemma_4_26b	350	0.597	0.069	87.6
claude_opus_4_7	407	0.571	0.071	101.9
mistral_small_creative	439	0.557	0.095	75.8
claude_opus_4_6	534	0.551	0.076	122.1

Spearman rank correlation between every pair of methods. Bayesian ELO row is the headline: it's uncorrelated with every LLM-judge method.


Behavioral repetition	-0.086	-0.094	-0.138	-0.146	-0.314	-0.086	-0.073	-0.143	-0.196	-0.082


Likert overall	1	0.912	0.61	0.841	0.855	-0.086	0.27	-0.143	-0.196	-0.09
F1 Likert mean	0.912	1	0.372	0.725	0.669	-0.094	0.281	-0.155	-0.161	-0.082
F2 Likert mean	0.61	0.372	1	0.686	0.59	-0.138	0.056	-0.13	-0.232	-0.142
F12 Likert mean	0.841	0.725	0.686	1	0.748	0.05	0.162	-0.02	-0.146	-0.068
F13 Likert mean	0.855	0.669	0.59	0.748	1	0.057	0.489	-0.314	-0.032	0.079
F1 binary rate	-0.086	-0.094	-0.138	0.05	0.057	1	0.085	-0.027	0.446	0.442
Flaw hunter mean	0.27	0.281	0.056	0.162	0.489	0.085	1	-0.073	0.14	0.185
Bayesian ELO	-0.143	-0.155	-0.13	-0.02	-0.314	-0.027	-0.073	1	-0.282	-0.227
Behavioral unique-wr	-0.196	-0.161	-0.232	-0.146	-0.032	0.446	0.14	-0.282	1	0.985
Behavioral repetition	-0.09	-0.082	-0.142	-0.068	0.079	0.442	0.185	-0.227	0.985	1

Per-model multi-signal profile (failure rates, behavioral, flaw hunter, subjective, ELO)

claude_opus_4_6

Model: claude_opus_4_6
──────────────────────────────────────────────────────────────────────────────

FAILURE RATES (lower = better)
  Agency violations       4.5%  [±6.9%]  ██░░░░░░░░░░  44 probes
  POV/Tense violations    0.0%  [±5.2%]  ░░░░░░░░░░░░  33 probes

BEHAVIORAL METRICS
  Avg words                533.525  (population avg: 264.589)
  Prose quality (unique wr)  0.551  (population avg: 0.655) ↓
  Repetition score           0.076  (population avg: 0.049) ↑

FLAW HUNTER (100 - deductions, target-aware)
  Mean score                 40.9/100  (mediocre)
  Median score               42.0/100
  Fatal flaws/session        0.29
  Major flaws/session        6.82
  Top flaws:              purple_prose, recycled_description, narrating_emotions
  Sessions scored:             17

SUBJECTIVE (LLM-judge, rubric proxy)
  Engagement                  4.67/5
  Tone consistency            4.75/5
  Collaboration               4.46/5
──────────────────────────────────────────────────────────────────────────────

NOT IN COMMUNITY ARENA POOL.  MT-judge mean: 4.51

Strength:  Top-2 on agency respect (4.55/5)
Weakness:  High phrase repetition (0.076 vs population 0.049)

RP-Bench: Roleplay Quality Benchmark

Bayesian Bradley-Terry ELO from 1,857 clean (suspect-filtered) human pairwise votes. 95% CI is wide (~260 points) — the top tier is statistically tied. Frequentist columns from the 100-shuffle ELO for comparison.

Sonnet 4 holistic Likert scores per session. F1-F13 columns are means on the seeds that target each failure mode. Lower 'Avg fail rank' = better cross-mode reliability.

Strict 100-point deduction rubric. Mean ~36, median ~42 across all sessions — the methodology forces lower scores than the Likert. Fatal/session column is the rate of -15 deductions; high values flag catastrophic single sessions.

Quality per dollar at OpenRouter prices (60/40 input/output blend, $/1M tokens). DeepSeek V4 Flash is 281× more cost-efficient than Opus 4.7 on flaw hunter for marginal quality difference.

Pure prose statistics across all 3,569 model-generated responses. Cannot be gamed by judge taste. Length-biased — compare within tiers, not across.

Spearman rank correlation between every pair of methods. Bayesian ELO row is the headline: it's uncorrelated with every LLM-judge method.

Per-model multi-signal profile (failure rates, behavioral, flaw hunter, subjective, ELO)

claude_opus_4_6