JevBench and JevArena have different purposes
JevBench describes a community benchmark for models that judge a state against a bounded rubric. It explicitly distinguishes the benchmark from an official TypeSafe endorsement. Inspect its release, task set, scoring and submitted model configuration before citing a result.
JevArena lets users compare Jev with another judge and inspect answers, latency, cost and available confidence data. A user-submitted arena case can reveal a failure or preference; it is not automatically a representative sample of your workload. There are also game projects with similar names, so verify the repository owner.
Keep comparisons on the same task and conditions
Use the same states, rubrics, answer definitions, labeled test set and preprocessing. Record versioned model IDs or checkpoint hashes instead of comparing moving aliases as if they were fixed models. Keep tuning examples separate from held-out evaluation.
For local models, record device, runtime, batch size, numerical precision and cold versus warm conditions. For APIs, record endpoint, request size, concurrency, retries and network conditions. Compare successful and failed requests together rather than silently excluding failures.
Accuracy and confidence are not the same metric
| Measure | What it answers |
|---|---|
| Per-class errors | Which labels or outcomes does the model confuse? |
| Calibration | How do emitted probabilities relate to observed outcomes across examples? |
| Coverage and review rate | How much work proceeds automatically, and what remains for review? |
| End-to-end success | Does the complete application perform the intended task? |
| Latency distribution | What are typical and slow-path request times, including retries? |
| Cost per completed task | What do inference, infrastructure and review consume together? |
The official System One explanation says calibration is measured across groups and does not guarantee an individual answer. The confidence documentation also distinguishes answer confidence from option probabilities. Keep those quantities separate in an evaluation.
Tune thresholds without hiding the tradeoff
A stricter review policy can improve accuracy among automatically processed cases by declining more work. Report both automatic coverage and errors. For independent tags, tune each tag according to its own false-positive and false-negative consequences rather than borrowing one threshold for every question.
Include ambiguous examples, non-English content relevant to your users, malformed inputs and cases outside the answer taxonomy. The Jev 1.13 notes identify failures involving literal wording, arithmetic, indirection and adversarial state; those are useful test categories.
JEPA, diffusion experiments and AGI claims
LeJudge describes adding natural-language constraint judgments to JEPA world-model planning. That is a composition of components, not evidence that Jev itself is a JEPA world model. Local decision architectures and diffusion-related projects likewise need their own implementation and evaluation evidence.
A broad label such as “AGI” is not established by a typed-decision benchmark or a successful game replay. Ask for the concrete task, available inputs, tested generalization and documented failure cases. Distinguish an author's architecture hypothesis from official model documentation.
What makes a result reusable?
A reusable result names the data, model, rubric, configuration, exclusions and method, and provides enough artifacts to reproduce the scoring. Read those artifacts before repeating a headline number. This guide provides evaluation criteria; JevHunt has not reproduced the community leaderboard scores or measured a model-speed comparison.