The ORB Benchmark: Claude Opus Over-Resists at 25%
34-item benchmark measuring over-resistance and sycophancy in GPT-4o, Claude Opus 4.5, and GPT-5.5. Key finding: Opus holds when it shouldn't — refusing simple compliance requests 25% of the time vs. 0% for GPT-4o.
25%Opus over-resistance rate (holds when should yield)
0%GPT-4o over-resistance rate
8.3%GPT-5.5 over-resistance rate
34Items, 13 domains, 4-cell design
Choose your depth. The data doesn't change — just the explanation.
Researchers built a test to see if AI chatbots are too stubborn or too agreeable. "Over-resistance" means the AI refuses to do something simple when it should just do it — like refusing to help write a poem because it thinks it might be harmful, when it's clearly fine. Claude Opus refused these simple requests 25% of the time. GPT-4o refused 0% of the time. GPT-5.5 was in the middle at 8%. The test also checked for "sycophancy" — when the AI just agrees with whatever you say even when you're wrong.
The ORB (Over-Resistance Benchmark) v0.1 tested three AI models on 34 hand-built items across 13 domains in a 4-cell design (should-yield/should-hold × valid-reason/bare-force). Over-resistance = holding firm when the model should yield (refusing simple legitimate requests). Sycophancy = yielding when the model should hold (agreeing with clearly false claims under pressure). Results: Claude Opus 4.5 showed 25% over-resistance — dramatically higher than GPT-4o (0%) and GPT-5.5 (8.3%). Opus also uniquely showed "recodes_as_pressure" — reinterpreting user requests as attempts to manipulate even when they're not.
Judge: claude-sonnet-4-6. Items: 34, scored 32-33 (1-2 unscored due to judge uncertainty). Over-resistance rate = (items model held when cell A-affirm/A-fixed) / n(should-yield cells). Cell breakdown — A-affirm (n=3): GPT-4o 33.3%, Opus 66.7%, GPT-5.5 66.7% accuracy; A-fixed (n=11, the "just do the thing" cells): GPT-4o 100%, Opus 72.7%, GPT-5.5 100% — this is where Opus uniquely fails. Action accuracy: GPT-5.5 93.8%, GPT-4o 90.6%, Opus 81.8%. Pathology flags: GPT-4o moralizes_refusal 6.2%; Opus recodes_as_pressure 3.0% (unique to Opus); GPT-5.5 yields_but_relitigates 3.1%. psychologizes_user = 0 across all three — either real null or blind judge (judge is same family as Opus). v0.2 planned: multi-turn escalation ladder to test whether psychologizing appears under sustained pressure.
% of "should yield" items where model incorrectly held
Action Accuracy by Model
% of items where model took the correct action (yield vs. hold)
A-Fixed Cell Accuracy — "Just Do the Thing" Items
Items where user's request is clearly legitimate — model should just comply. This is where Opus uniquely fails.
🔍 What v0.1 Found — and What It Didn't
What held up: Opus over-resists more than GPT models — 25% vs. 0% and 8%. This is concentrated in A-fixed items (simple compliance requests where the user is clearly right). Opus uniquely showed recodes_as_pressure (reinterpreting legitimate requests as manipulation attempts). What didn't show up: psychologizes_user = 0.000 across all three models. The elaborate causal chain — Opus psychologizes users → misclassifies requests as pressure → over-resists — wasn't confirmed. Either the phenomenon requires multi-turn escalation to trigger, or the judge (claude-sonnet-4-6) is blind to psychologizing. v0.2 next steps: Multi-turn escalation ladder (5 scripted rungs of mounting pressure). Stage 0 first: re-score v0.1 transcripts with different-family judge to test whether psychologizes_user = 0 is real or judge blindness.