An experimental protocol for evidence-aware agent handoffs, bounded worker continuation, and TypeSafe/Jev-assisted review, with reproducible evaluation.
Evaluation & benchmarks — page 7
Evaluation harnesses, benchmarks and graders. Jev scores or chooses between candidate outputs with an explicit rubric.
320 active projects match this task, ordered by traction.
Judge agent steps — ok / retry / escalate / stop via TypeSafe Jev
Reproducible benchmark for TypeSafe AI's Jev on agent tool-call risk classification: accuracy, latency, and whether the confidence score is worth routing on.
Jev-powered browser MCP for LLM agents — ~300ms decisions, no LLM tokens in the loop. Benchmark vs Playwright MCP included.
Benchmarks Jev against other evaluation models in games with explicit states, legal actions, and measurable outcomes.
A small second eval for shadcn-ui/lint that uses TypeSafe's Jev to judge the linter's own output.
TypeSafe's Jev model plays Snake live in the browser, with a headless eval and a self-improvement loop
Guardrail + model router for LLM gateways on TypeSafe's Jev (System One model), with an independent accuracy/calibration/latency evaluation. Stdlib Python.
Daily LLM benchmark on Poople (poople.io): one-shot word ladders to POOP, scored against provable optimal, accuracy vs cost via OpenRouter
Benchmark for silent omissions in AI coding-agent patches: 310 labeled instances from 8 Python libraries, a deterministic AST detector, 4 LLM/probabilistic judges, pre-registered rules and paired cluster-bootstrap CIs. Includes the results that went against the thesis.
1
Evaluation System for DG Fundraising Cause
Open-source adaptive evaluation for LLM, RAG and agentic systems. System One judges first, LLMs only when necessary.
Jev (TypeSafe AI) as a System One decision layer for a multilingual EN/UZ/RU RAG helpdesk, with an independent benchmark
0
Local-first hybrid RAG: traditional vs Jev-style (System One) pipelines over your own documents — with a built-in benchmark lab (6 scenarios, 48 questions, independent LLM judge). Hybrid won +8.4pp correctness at identical cost.
Model Context Protocol (MCP) server & OpenCode skill for San Francisco Early Learning For All (ELFA) preschool search, voucher calculations, and TypeSafe Jev System One evaluation
Use Jev instead of an LLM for RAG decisions: relevance grading, query routing, hallucination checks and LLM-as-a-judge evals. Faster, cheaper, measured first.
Version-aware evidence decisions for RAG assistants using Jev, with inspectable routing, bounded recovery, and reproducible evaluation.
A quality gate for AI-generated requirements, powered by Jev (TypeSafe System One). Routes each requirement to PASS / REVIEW / BLOCK with calibrated probabilities. Real benchmark vs LLM: 0 false blocks, 9x cheaper.
Run only the tests your change touched: import graph + ownership rules first, Jev decides the ambiguous rest, fail-safe to the full suite. Replay benchmark with real ground truth.
0
Open emulator of TypeSafe's Jev API on open models, plus a paired benchmark of local and hosted models against Jev
A work ledger for coding agents that verifies what they report. Plans, tasks, claims and handoffs live in SQLite, and a Jev-compatible judgment model checks briefs, done reports and replies against them. Runs as a Claude Code and Codex plugin, with hosted Jev, open Jev servers, Laya or Jev-Style as the judge.
0
Bobcat: a typed-decision model. State in, Choice / Noul / Score out, with a probability for every answer you name; no generated text. Compiler, server, training and evaluation code for Bobcat 1.1 and Bobcat Flash 1.1.
0
JEV Studio is a small Next.js evaluation lab for turning natural-language input into structured signals with the Typesafe SystemOne API.
Local judge first, cloud Jev second: a privacy-first request router (llama.cpp + TypeSafe Jev)