scienthoon/jev-ood-calibration
VerifiedResearchDocumented
jev-ood-calibration: Independent calibration test of TypeSafe's Jev on a task it cannot have seen. Explore the Jev references, Python source code and similar projects.
Repository description: Independent calibration test of TypeSafe's Jev on a task it cannot have seen: 900 rule-generated support tickets (choice / score / boolean) plus 3 public benchmarks via Vercel AI Gateway. Raw responses, ECE with noise floor, temperature refit, per-type sign of miscalibration. Reproducible for ~$0.06.
Clone the repository and follow its README for install and usage. JevHunt has not run this project.
git clone https://github.com/scienthoon/jev-ood-calibration.gitStars measure the whole repository, including work unrelated to Jev.
Documented records the linked documentation or source. JevHunt has not independently run or benchmarked this project.
All calls through Vercel AI Gateway (model id `typesafe-ai/jev`; the `typesafe-ai/jev-latest` id in the AI SDK docs returns "Model not found" on the Gateway), AI SDK 7.0.107 `experimental_evaluate`, `zeroDataRetention: true`, 2026-09-19. 3,721 public-benchmark items + 900 synthetic items, 0 failed calls. The Gateway does not expose a model version; every response resolves to canonical slug `typesafe-ai/jev` and declares `rounding: {probabilityDecimals: 2, scoreDecimals: 2}`, which is where the 0
Evidence commit: 914d87ab16517f14e833cff63d73b51235b14bd6
Discovered through: awesome-jev, typesafe-field-guide.
How we review →Suggest a correction · Claim this listing · Submit a project