# Benchmark: "System 1" record screening, Jev vs Claude vs rules (SYNTHETIC data)

> **DỮ LIỆU TỔNG HỢP / SYNTHETIC.** All 300 records are fictional (names, communes, lô numbers, documents, volumes, dates). Ground truth comes from an expert rubric written *before* generation, not from field checks. Run date: 18/10/2026. All numbers are in `benchmark_jev.json`. Chart: `web/benchmark_jev_chart.png`.

## Question
Can a fast "System 1" decision model (TypeSafe **Jev**) triage household planted-acacia harvest records ("1 triệu hồ sơ/năm") into **cần kiểm tra hiện trường** (escalate) vs **xếp lưu có lý do** (close)? How does it compare with Claude Haiku 5.5, Claude Sonnet 5.5, deterministic rules and a Jev→Sonnet cascade?

## Method
- **Rubric** (`data/bench_jev/rubric.json`, `data/scripts/jev/rubric.py`, frozen before any record was generated). It is built from critique_law.md §D.4. A record is escalated if it shows any of these signs:
  - **D1** change outside the lô > 0,30 ha (> 0,10 ha if the adjacent lô is natural protection forest);
  - **D2** m³/ha above an age bound (90/120/145/165/180/195/210 m³/ha for ages ≤4…≥10);
  - **D3** max log diameter above an age bound (25/30/35/40 cm) or non-acacia logs;
  - **D4** harvest start (earliest of the declared start and the start of the Hệ 1 detection window) before the Hạt received Mẫu 08;
  - **D5** no Mẫu 08 but > 0,30 ha changed inside the lô;
  - **D6** a suspicious ranger note (an unverified suspicion still counts).

  Tie-breaks:
  - **T1** strict ">" on point estimates;
  - **T2** precautionary window start;
  - **T3** D1/D5 count as explained only by a field check with a biên bản number, and never next to natural protection forest;
  - **T4** negated notes are not signs;
  - **T5** units.

  The bounds are an illustrative expert choice, not an official standard.
- **Data** (`gen_records.py`, seed 20261018): 300 records, **85 escalate (28,3%)**, **49 ambiguous (16,3%)**. The ambiguous ones are cases at a threshold, windows that straddle the filing date, and values exactly equal to a bound; the tie-break rules decide their labels. Each record has a semi-structured part (Mẫu 08, Mẫu 01, Hệ 1 overlay, adjacent lô) and a free-text Vietnamese ranger note. Realistic format noise: m² vs ha, decimal commas, 4 date formats, m³/m3/"khối", 3 ways of writing diameters, and start dates that appear only in notes. Note types: paraphrased red flags, hedged suspicions, negated distractors, verified explanations (with a biên bản number), unverified explanations.
- **Split:** a stratified split of 100 calibration and 200 test records (57 escalations in test), fixed before any model ran. **Every threshold was tuned on calibration only**, with one rule fixed in advance: minimise 5·FN + FP (a missed escalation costs 5× a needless field trip). All headline metrics are on the test split.
- **Every contender got the same rubric text.** The task therefore measures *applying a written policy* (arithmetic, dates, Vietnamese notes), which is what an agency would do.

Contenders:

| Contender | How |
|---|---|
| Rules | Regex parser over the record **text** plus exact rubric checks. Note keywords taken literally from the policy wording and frozen after the first run. |
| Jev raw | `state = {policy, record}`; Noul "cần kiểm tra hiện trường?", Choice "lý do chính" (7 options), Score "mức rủi ro" (5 levels). One call per record. Run on jev-1.13.0 (`jev-latest`) and jev-1.14.0 (`jev-preview`). |
| Jev atomic | Same state; 7 Nouls (D1–D6, T3), combined in code (noisy-OR), plus a variant with a logistic head trained on calibration (the vendor's "composite scoring" pattern). |
| Jev + code (hybrid) | Code does the numbers and dates (rules parser). Jev reads only the ranger note (2 Nouls: red flag? verified explanation?). This is the vendor-documented "keep arithmetic in code" pattern. |
| Haiku 5.5, Sonnet 5.5 | `claude -p … --model haiku/sonnet --output-format json --tools "" --strict-mcp-config --no-session-persistence`. **10 records per call** (30 calls each; cheaper and less overhead per record). They return escalate, a stated p, a reason and a risk level. CLI defaults (thinking on). |
| Cascade | Jev raw 1.13 first: p < τ_low → close, p ≥ τ_high → escalate, otherwise use Sonnet's answer. The τ pair minimises the share sent to the LLM on calibration, subject to FN ≤ Sonnet-alone and errors ≤ Sonnet-alone + 1. |

## Results (test split, n = 200, 57 true escalations)

| Contender | Acc. | Prec. | **Recall** | F1 | Missed (FN) | FP | AUC | ECE | p50 latency / record | $ per 1.000 (API list) | $ per 1 M |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Rules (full rubric) | 0,975 | 0,98 | 0,93 | 0,955 | 4 | 1 | – | – | 0,05 ms | ≈ 0 | ≈ 0 |
| Jev 1.13 raw @0,5 | 0,325 | 0,30 | 1,00 | 0,458 | 0 | 135 | 0,857 | 0,593 | 0,42 s | 0,098 | 98 |
| Jev 1.13 raw @τ=0,88 | 0,530 | 0,37 | 0,95 | 0,535 | 3 | 91 | 0,857 | 0,593 | 0,42 s | 0,098 | 98 |
| Jev 1.14 raw @τ=0,69 | 0,555 | 0,39 | 0,98 | 0,557 | 1 | 88 | 0,902 | 0,479 | 0,42 s | 0,098 | 98 |
| Jev 1.13 atomic @τ | 0,835 | 0,65 | 0,89 | 0,756 | 6 | 27 | 0,915 | 0,519 | 0,40 s | 0,097 | 97 |
| Jev 1.13 atomic + LR head | 0,595 | 0,41 | 0,93 | 0,567 | 4 | 77 | 0,870 | 0,047 | 0,40 s | 0,097 | 97 |
| **Jev 1.13 + code (hybrid) @0,5** | 0,970 | 0,90 | **1,00** | 0,950 | 0 | 6 | 1,000 | 0,096 | 0,40 s | **0,035** | **35** |
| Jev hybrid @τ=0,95 | 0,990 | 1,00 | 0,96 | 0,982 | 2 | 0 | 1,000 | 0,096 | 0,40 s | 0,035 | 35 |
| Claude Haiku 5.5 (batch 10) | 0,995 | 0,98 | **1,00** | 0,991 | 0 | 1 | 0,999 | 0,129* | 3,6 s (single call 6,9 s) | 0,50 (CLI 0,61) | 505 |
| Claude Sonnet 5.5 (batch 10) | 0,995 | 0,98 | **1,00** | 0,991 | 0 | 1 | 0,994 | 0,058* | 1,5 s (single call 6,0 s) | 3,28 (CLI 5,49) | 3.278 |
| Cascade Jev raw → Sonnet | 0,960 | 0,92 | 0,95 | 0,931 | 3 | 5 | – | – | mix | 2,06 (CLI 3,39) | 2.064 |

\* LLM p is a stated (verbal) probability. Jev ECE uses its native Noul probability. "Missed" = true escalations closed.

- **Cascade:** with raw Jev as the first tier, **60% of test records still had to go to Sonnet** (58% on calibration), because raw Jev puts almost every record above 0,88. The Jev-atomic+LR cascade sends 50%. Neither beats Sonnet alone: 3–4 escalations were auto-closed.
- **Jev raw calibration** (test, reliability table in JSON): 137 of 200 records sit in the 0,9–1,0 bin with mean p 0,95, yet only 39% of them are true escalations. Records at p 0,6–0,9 escalate at 3–10%. Its ranking carries information (AUC 0,86–0,90), but its probabilities are not calibrated *on this task*. The hybrid design is well behaved: everything below p 0,7 was a true close, and every record ≥ 0,9 was a true escalation.
- **Where raw Jev fails:** per-sign recall (test) at τ is D1 0,95, D3 0,86, D4 0,83 and D6 1,00, with heavy over-flagging of clean records. Atomic per-sign AUCs were high (D1 0,99, D2 0,99, D5 1,00, D6 1,00, D4 0,88, all 300 records), so Jev "sees" the signs. Combining them and reasoning over numbers and dates in one long policy is where it breaks, consistent with TypeSafe's own "jaggedness" page (math, dates, literal reading). Reason agreement on true escalations: Jev 0,77–0,81, rules 0,93, Haiku 0,95, Sonnet 1,00.
- **Rules:** perfect on every structured sign (recall 1,00 for D1–D5). They missed 4 of 8 note-only escalations (paraphrases such as "không cho xem hợp đồng", "vỏ sần, không giống gỗ keo"). They also wrongly escalated 1 record: a verified fire biên bản that contains "không phát hiện chặt hạ" tripped the negation filter. Accuracy on records with a substantive note was 0,81, vs 0,96 for the hybrid and 1,00 for the LLMs.
- **Throughput at concurrency 4:** Jev 7,5–8,7 records/s, so 1 M records take ~37 h. Haiku 1,0 rec/s with batches of 10 (thinking ~7.800 tokens per batch), ~276 h. Sonnet 2,5 rec/s, ~112 h. Rules: 1 M in < 1 minute on one core. No Jev 429s were seen. Rate limits are per organisation and shown only in the console.
- **Consistency (30 test records re-run):**
  - Jev raw 1.13: 1 decision changed at 0,5 and 0 at τ; mean |Δp| 0,022, max 0,18. Jev is **not bit-deterministic**.
  - Jev 1.14: 1 changed at 0,5 and 4 at τ=0,69.
  - Jev atomic: 3 changed at τ.
  - Jev hybrid: 0 changed.
  - Haiku: 0 changed.
  - Sonnet: 1 changed (the re-run used different batch composition).
  - Rules: 0 changed.

### Legal-reasoning check (Vals AI claim)
The questions are open-book multiple choice, with a digest of provisions paraphrased from critique_law.md §C (TT 26/2025, NĐ 146/2026, BLHS Đ243, Luật TTNT, EUDR) in the state.
- **Lookup set** (18 questions, e.g. "khoản nào của Đ23 NĐ 146/2026…"): Jev 1.13 **18/18**, Jev 1.14 18/18, Haiku 18/18, Sonnet 18/18. Not discriminative.
- **Multi-step set** (12 questions, added after seeing that result; total 30, above the planned 10–20). It covers transitional law (Đ41), area band × forest type × ×2, organisation ×2 fines, a negated stem, roles, EUDR Art. 13(2) and AI-law dates. Jev 1.13 **11/12** and Jev 1.14 11/12. The miss was the same for both: H10, a date question (Luật TTNT Đ35 transition), answered with p 0,76. Haiku 12/12 and Sonnet 12/12. (Haiku copied the example id format "L01"; its answers were mapped by position, see JSON.)
- **Verdict:** this small test **does not reproduce** Vals AI's LegalBench gap (Jev last of 12). Jev's only error is a date calculation, which matches the vendor's documented weakness. The deck line "Hệ 1 để sàng lọc, không để kết luận pháp lý" should rest on **Luật TTNT Đ4 k2 / Đ27 k2** and the Vals AI result. It should not cite this test as evidence that Jev is bad at law.

### Experiment B: Jev 1.14 on the 100 satellite tile pairs (image preview)
Jev 1.14.0 accepts images with the header `TypeSafe-Danger-Zone-Images: true`. Images must be 280–1568 px; the docs label the feature not production-ready. The reference is Hansen loss 2019–2025 per tile, as defined by the satellite agent (`data/bench/tiles.json`, `data/out/benchmark.json`).

| | Acc. (≥10%, prevalence 0,76) | F1 | Acc. (≥20%) | MAE fraction | Spearman | p50 latency | $ / 1.000 tiles |
|---|---|---|---|---|---|---|---|
| Jev 1.14 Noul @0,5 | 0,77 | 0,85 | – | – | 0,27 | 0,77 s | 0,094 |
| Jev 1.14 Score → fraction | 0,79 | 0,88 | 0,43 | 0,136 | 0,24 | 0,77 s | 0,094 |
| LightGBM Hệ 1 (other agent) | 0,86 | 0,92 | 0,80 | 0,048 | 0,90 | 0,03 s | ≈ 0,01 |
| Vision LLMs (other agent: Haiku / Sonnet / Opus) | – | – | – | 0,107 / 0,084 / 0,110 | 0,26 / 0,32 / 0,30 | 2,3–5,3 s | 0,34 / 3,3 / 10,3 |

"Always yes" scores 0,76 on the ≥10% label, so Jev's image answers are essentially the base rate. Its ranking is as weak as the vision LLMs' and far below the pixel model.

## Headline claims the data supports
1. **On this synthetic task, raw Jev is fast and cheap but not usable as a stand-alone screener.** It runs at 0,4 s and ~$98 per million records, but reaches 53–56% accuracy at a calibrated threshold, and its "0,95" means escalate only 39% of the time. In the vendor's own recommended pattern (code does the numbers, Jev reads the free-text notes) it reaches **recall 1,00, accuracy 0,97, at ~$35 per million**.
2. **Rules plus a small model on the free text is the right Tier 1. The LLM is not needed on every record, but here it was the most accurate.** Rules alone: 0,975, with failures only on paraphrased notes. Haiku and Sonnet: 0,995 with no misses, at $505 / $3.278 per million and 4–9× slower per record than Jev (batched). At 1 M records per year every option is affordable. The real differences are latency, auditability and where the data goes.
3. **Thresholds must be calibrated on local data.** The default p ≥ 0,5 for raw Jev flags 96% of records. A Jev-first cascade still sent 50–60% to the LLM and auto-closed 3–4 true escalations. Jev's legal answers were 29/30 correct with a digest, so this test does not confirm the Vals LegalBench gap. The "not for legal conclusions" principle comes from Luật TTNT Đ4 k2 / Đ27 k2.

## Honest limitations
- The records are synthetic, written by the same author as the rubric, the rules and the prompts. Real records (scans, OCR, missing fields, inconsistent forms) are messier and would hurt the rules most.
- Truth is defined by the rubric, so "accuracy" means agreement with that rubric, not legality.
- One prompt design per contender (three question designs for Jev). Jev results may change with better question engineering. The vendor states English is its strongest language.
- LLMs were batched 10 per call. Neighbouring records may influence answers.
- Prices: Jev's is a vendor list price that may be subsidised. Claude API-only cost uses list prices minus the estimated ~3,6k-token CLI overhead (with tools disabled; the earlier "16k" figure applies to a default CLI session). Thinking tokens are billed as output. Measured CLI cost is also reported.
- Test n is 200 with 57 escalations, so a 1–2-record difference is noise. Jev is not deterministic between runs.
- The legal check is small and open-book, and its harder set was added after seeing the first results.

## Reproduce
Scripts are in `data/scripts/jev/`:
- `rubric.py`, then `gen_records.py`, then `make_split.py`;
- `run_jev.py {raw|atomic|hybrid} <model>`;
- `run_llm.py {haiku|sonnet}`;
- `run_legal.py` and `run_legal_hard.py`;
- `run_tiles.py`;
- `analyze.py` and `chart.py`, run with `data/.venv/bin/python`.

Work files are in `data/bench_jev/`. The Jev key is read at runtime from `.secrets/jev.env` and is never written anywhere. API usage: 1.493 Jev HTTP requests (no 429s; 2 connection errors and 1 deliberate 400 image-size test) and 82 Claude CLI calls (total CLI cost $2,16), concurrency ≤ 4.