A benchmark suite for evaluating user simulators.
SimEval-IR evaluates simulated search and conversational sessions along three axes: behavioral realism, system effectiveness, and tester reliability, under one schema across datasets.
Session · search
Tquery_issuedneural retrieval models
clickrank 1 · dwell 34s
clickrank 3 · dwell 12s
query_issuedreranking vs bm25
0.142
JSD action types
0.089
Session length KS
0.71
Kendall τ
Three evaluation objectives.
Behavioral realism
Measure how closely a simulator reproduces real user behavior, from action distributions to session shape.
JSDFréchet distanceRealism classifier
System effectiveness
Evaluate systems with session-aware effectiveness, accounting for how a session unfolds, not just a single query.
Session nDCGExpected global utility
Tester reliability
Check whether a simulator ranks systems the way real users would, and weight testers by how much they agree.
Kendall τSpearman ρRATE
Two session types, one schema.
T
Traditional search
Query-based sessions with rankings and clicks.
- Query reformulation sequences
- SERP views and click patterns
- Dwell times and navigation
- Relevance judgments when available
C
Conversational
Multi-turn dialogues with user and system turns.
- User and system utterances
- Turn-based structure
- Retrieval within conversations
- Recommendations and feedback
The framework at a glance.
The schema, objectives, metrics, and integrations in a single view.
Behavioral realism
- JSD
- Fréchet distance
- Realism classifier
System effectiveness
- Session nDCG
- Expected global utility
Tester reliability
- Kendall τ
- Spearman ρ
- RATE
Session types
- TTraditional search
- CConversational
Integrations
- SimIIRLoad logs from SimIIR simulations directly.
- PyTerrierConvert runs and evaluate inside IR pipelines.