Methodology
How PsychBench surfaces behavioral differences in AI reasoning through adversarial poker gameplay.
Why Poker?
Poker is uniquely suited to revealing how AI models reason under pressure. Unlike traditional benchmarks, poker requires simultaneous deployment of skills that expose genuine behavioral differences:
Incomplete Information
Making decisions with hidden cards tests reasoning under uncertainty
Multi-Agent Dynamics
Competing against multiple opponents reveals social modeling capabilities
Risk Assessment
Bet sizing and fold decisions expose risk tolerance and confidence
Temporal Reasoning
Strategy must adapt across hands as blinds escalate and stacks change
Study Design
Public Rating Mode
The public leaderboard emphasizes 1v1 outcomes and normalized ratings so models with different evaluation coverage can still be compared clearly.
Heads-Up Mode
Models are evaluated in 10-game 1v1 matchups. Major benchmark refreshes can use broad round-robin coverage; new releases use adaptive placement against opponents chosen to be maximally informative for the rating.
Heads-Up Elo
The heads-up leaderboard is ranked by a Bradley-Terry Elo score, not raw win totals. Raw wins are useful context, but they can be misleading when models have different numbers of completed games. Elo estimates relative strength from pairwise outcomes, so beating a stronger opponent moves a model more than beating a weaker one.
For newly released models, the target is the same latent strength estimate we would get from a full matrix, without spending games on pairings that add little information. We start with anchor opponents, refit the rating after each block, then choose the next opponent where the estimate has the most to learn.
Placement runs continue until fixed evidence gates pass: enough games, enough distinct opponents, opponents that bracket the estimate or anchor the top/bottom edge of the field, local matchups near the estimate, and a confidence interval below the publication threshold.
Input
Completed 1v1 matchups, including full-matrix and placement runs, contribute pairwise outcomes to the internal rating fit.
Estimate
A Bradley-Terry model fits one strength value per model from the observed connected pairwise graph, then converts those strengths to an Elo-style scale.
Uncertainty
Confidence intervals and opponent coverage show how much evidence supports each rating. Incomplete or failed runs are excluded from the fit.
The math behind the ranking
Each model gets a latent strength parameter, written as theta. The Bradley-Terry model asks: given two models with strengths theta_i and theta_j, how likely is model i to beat model j?
The fitted theta values maximize the likelihood of the observed pairwise results. We then center the field at 1500, so 1500 means median strength within this evaluated model set. A 100-point Elo gap corresponds to about a 64% expected win rate for the higher-rated model against the lower-rated one. A 200-point gap corresponds to about 76%.
Public rankings show normalized ratings, confidence ranges, and coverage status; raw per-model records are retained internally for audit and calibration.
How We Built the Classifiers
Iterative human-in-the-loop refinement to establish trustworthy psychological labels
Multi-Rater Classification Pipeline
4 diverse LLM raters independently classify each decision, majority vote determines final label
Why Trust These Labels?
Inter-rater reliability metrics ensure consistent, reproducible classifications
Diverse Raters
4 LLMs from 4 providers reduce single-model bias.
Majority Consensus
Labels only applied when 3 of 4 raters agree.
IRR Metrics
Statistical measures for agreement beyond chance.
ECAAMS Framework
19 dimensions across 6 psychological axes
Limitations & Caveats
What We Measured
19 psychological dimensions that capture how AI models express reasoning.
Tap any dimension code to see its definition
Emotion
3 dimensionsHow often feelings surface in reasoning
"I feel confident about this play..."
Positive vs negative emotional framing
"This is exciting!" vs "This is worrying..."
Attempts to manage emotional state
"I need to stay calm here..."
Cognition
3 dimensionsExplicit logical analysis and calculation
"Calculating pot odds: 3:1..."
Thinking about own thought process
"Let me reconsider my approach..."
Internal debate or uncertainty
"On the other hand..." or "But maybe..."
Action
4 dimensionsFocus on taking action vs passive observation
"I need to make a move here..."
Taking ownership of decisions
"I decide to fold" vs "Folding is optimal"
Decisiveness in committing to choices
"I will raise" vs "Maybe I should..."
Reasoning matches actual action taken
Saying "I should fold" and then folding
Arousal
3 dimensionsPhysical sensations like gut feelings
"Something feels off about this..."
Elevated energy or excitement
"This is intense!" or "Heart racing..."
Awareness of physical state
"I need to take a breath..."
Meaning
3 dimensionsReferences to self-concept or role
"As a tight player, I typically..."
Judging own performance
"That was a good/bad call..."
Connecting decisions to a larger story
"This fits my strategy of..."
Social
3 dimensionsModeling opponent beliefs and reasoning
"They probably think I have..."
Judging or evaluating opponents
"This player seems aggressive..."
Framing as competition or conflict
"I need to beat them here..."
Models Evaluated
Game Configuration
Starting Conditions
- 60,000 chips per model
- No-Limit Texas Hold'em rules
- Full game state provided each decision
- Extended reasoning enabled for all models
- Reasoning summaries captured when exposed by the provider
Blind Schedule
Before each hand, two players must post mandatory bets called "blinds." These forced bets create action and increase over time to speed up the game.
Higher blinds = bigger pots = more pressure to make decisions
Reasoning Settings
We run models with reasoning enabled when the provider supports it. Effort labels are provider-specific, so this records the requested setting rather than treating every model's thinking budget as identical.
| Model group | Setting | Notes |
|---|---|---|
| GPT-5.2 / GPT-5.4 | medium | OpenAI reasoning effort, with detailed summaries requested. |
| GPT-5.5 | high | Raised from the default path so reasoning summaries are captured reliably. |
| Claude Opus 4.5 | 4096-token budget | Legacy Anthropic extended-thinking budget. |
| Claude Fable 5 / Opus 4.6 / 4.7 / 4.8 | high | Anthropic adaptive thinking. Fable and Opus 4.7+ use summarized thinking output. |
| Gemini 3 / 3.1 / 3.5 | HIGH | Google thinkingLevel. PsychBench medium/high map to HIGH. |
| Grok, DeepSeek, Kimi, Qwen, GLM | provider-native | Reasoning is enabled where supported; some providers do not expose a comparable effort slider. |
Actual reasoning-token counts are reported separately in the reasoning analytics views.
Bluff Classification
We classify each aggressive action (raise) as bluff, semi-bluff, or value based on hand equity and street-specific thresholds. This method aligns with standard poker analytics definitions used by professional players.
Classification Definitions
Street-Based Equity Thresholds
Thresholds vary by street because later streets have more information and different risk profiles.
| Street | Bluff | Semi-Bluff | Value |
|---|---|---|---|
| Pre-flop | <25% | 25-50% | >50% |
| Flop | <20% | 20-50% | >50% |
| Turn | <18% | 18-45% | >45% |
| River | <15% | 15-55% | >55% |
River raises are more polar (bluff or value) because there are no more cards to come. Earlier streets allow semi-bluffs with drawing hands.