12 models (incl. Kimi K3) × 4 suites × 2 tiers, n=3 · + agent harness + Repo Prompt CE orchestration & planner sweep · median of 3 reps · via OpenRouter · 2026-08-11
Mean of all four suites (0–100). ◆ / outlined bar = runs locally on ≤24 GB (your P40 rig).
Code tasks only (N1 implement + N2 debug, both tiers). one-shot → agent (4-tool loop, runs the visible tests as feedback). Δ = agent − one-shot. Sorted by agent score. ▪ = runs on ≤24 GB.
The harness is an equalizer: models that can't one-shot hard code (gpt-oss-20b/120b, muse-glimmer) leap +50–65 when they can iterate against tests, while models that already one-shot well (kimi, deepseek) slip a little as the multi-turn loop adds under-scoping. qwen models mostly decline to drive the loop (call done early). Agent runs are noisier than one-shot (see the wide spreads); n=3, treat small gaps as ties.
The oracle/editor split, run natively through Repo Prompt CE on-device: a strong planner writes the plan, a cheap open model applies it.
Takeaway: the orchestration win is real and it's the oracle doing the heavy lifting — a strong planner turns a fuzzy bug report into an exact, apply-ready plan, and a 30B-class editor that couldn't touch this task alone applies it perfectly. Note: the open-model editor only wrote files in an interactive RepoPrompt session; the same agent completed headlessly twice without editing, so this arm needs a human at the keyboard for now.
Each model as the Repo Prompt oracle, producing the full N2 fix; graded on the hidden suite. Single run each.
| oracle model | fix score | |
|---|---|---|
| gpt-oss-120b | 100.0 | ✓ correct |
| kimi-k2.6 | 100.0 | ✓ correct |
| deepseek-v4-flash | 100.0 | ✓ correct |
| muse-glimmer-30b | 100.0 | ✓ correct |
| glm-5.2 | 83.3 | missed a bug |
Nearly every planner — including the local Muse Glimmer 30B — produces a fully correct fix. The lone miss is frontier GLM-5.2, which compared the tag name against "#e" instead of "e": a subtle nostr-specific slip a generic planner makes and a nostr-aware one doesn't.
Up = more accurate, left = cheaper. Log cost axis. local-capable (≤24 GB) cloud / bigger-iron.
Click a header to sort. Task columns shaded by score (darker = higher). sens = prompt sensitivity (T2 − T1): high positive = needs the engineered prompt; negative = the terse prompt worked better. −n = cells dropped (empty completion; see notes).
| model | runs on | acc | nostr | sec | N1 impl | N2 debug | S1 audit | S2 nostr-sec | T1 | T2 | sens | cost | pts/$ | drop |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| kimi-k3 | cloud | 91.7 | 90.3 | 91.7 | 83.3 | 100.0 | 95.8 | 87.5 | 94.8 | 88.6 | -6.2 | $0.6629 | 3,320 | |
| kimi-k2.6 | cloud | 91.3 | 89.7 | 87.0 | 91.4 | 100.0 | 96.1 | 77.8 | 90.0 | 92.8 | +2.8 | $1.0966 | 1,915 | −1 |
| deepseek-v4-flash | cloud | 85.4 | 83.8 | 83.6 | 90.5 | 83.3 | 89.6 | 77.5 | 84.6 | 86.1 | +1.5 | $0.0367 | 51,193 | −2 |
| deepseek-v4-pro | cloud | 84.7 | 83.7 | 84.2 | 78.6 | 91.7 | 87.5 | 80.9 | 79.2 | 90.1 | +10.9 | $0.3382 | 6,011 | |
| glm-5.2 | cloud | 83.3 | 79.0 | 83.3 | 66.7 | 100.0 | 96.3 | 70.2 | 81.0 | 85.5 | +4.5 | $0.311 | 6,428 | |
| gemma-4-31b-it | ▪ 24GB | 79.5 | 72.0 | 86.4 | 57.1 | 69.4 | 95.8 | 77.0 | 75.2 | 84.3 | +9.1 | $0.0965 | 15,653 | −4 |
| qwen3.5-397b-a17b | cloud | 75.0 | 68.7 | 84.9 | 57.1 | 76.7 | 95.0 | 74.7 | 71.7 | 78.0 | +6.3 | $0.3126 | 5,038 | −3 |
| qwen3.6-35b-a3b | ▪ 24GB | 69.5 | 60.0 | 90.9 | 48.8 | 47.2 | 97.9 | 84.0 | 68.4 | 70.5 | +2.1 | $0.245 | 6,808 | |
| muse-glimmer-30b | ▪ 24GB | 62.4 | 51.2 | 88.0 | 7.1 | 66.7 | 96.1 | 79.9 | 52.3 | 72.6 | +20.3 | $0.0729 | 20,543 | |
| gpt-oss-120b | ○ 96GB | 56.7 | 45.6 | 84.2 | 33.3 | 25.0 | 89.8 | 78.5 | 64.8 | 48.5 | -16.3 | $0.0144 | 94,500 | |
| gpt-oss-20b | ▪ 16GB | 33.3 | 16.5 | 62.5 | 8.3 | 0.0 | 83.8 | 41.2 | 30.5 | 36.1 | +5.6 | $0.0079 | 101,165 | |
| llama-3.3-70b-instruct | ○ 48GB | 25.9 | 17.7 | 44.0 | 7.1 | 8.3 | 50.4 | 37.6 | 25.4 | 26.4 | +1.0 | $0.016 | 38,850 |
Method: each cell is one completion, temperature 0.2, graded by a hidden vitest suite (N1/N2) or a recall×precision findings scorer (S1/S2). Median of 3 reps; whiskers on the bar chart show the min–max spread across reps. Treat gaps ≤5 pts as ties. Reasoning models were given a 30k-token budget with a 15k reasoning cap; kimi-k2.6 ignored the cap on the long N1-T2 prompt and is missing that one cell. Local flag = fits ≤24 GB VRAM per the run matrix. Costs are OpenRouter-reported per request.