Jev vs frontier LLMs on thousands of real shipments

In short

• On 1,222 real US import shipments, Jev identified the mode of transport with 98.9% accuracy. Six frontier-model setups scored 99.4%: a small gap, but a real one.

• Jev cost $0.35 for the whole test, 74 to 660 times less than the frontier setups, and answered in a median 234 ms.

• Used together, with Jev answering when it is at least 90% sure and Gemini 3.1 Pro taking the other 6% of shipments, the two scored 99.3% for 13 times less than Gemini 3.1 Pro alone, which scored 99.4%.

• Most of the work was in the inputs: an input limit, PDFs one API rejects, and OCR. Without trimming long shipments to fit Jev’s input limit, it would have scored 1.1 points lower.

The task

Every US import entry declares how the goods arrived: by vessel, air, truck or rail. It is Block 9 on CBP Form 7501, and a customs broker works it out for every shipment from whatever the shipper sent: bills of lading, air waybills, commercial invoices, arrival notices and spreadsheets, often scanned and sometimes not in English.

The paperwork can mislead. Freight that crossed the Pacific by ship and entered the US by rail from a Canadian port is a rail entry, even though its bill of lading names a vessel. Amari’s production definition spells out those rules, and every model in this test was given it.

It is a bounded choice among four answers, which is the shape TypeSafe built Jev for. Jev does not write text. It answers typed questions and returns a probability for each option.

Why it matters beyond customs

Mode of transport is one of the first facts every part of the supply chain needs about a shipment. Freight forwarders book and price by it, and issue a different document for each: an air waybill or a bill of lading. Warehouses plan inbound work around it: an ocean container needs drayage and unloading, while air freight arrives on loose pallets. Insurers price cargo cover by it, and emissions reporting calculates freight emissions per mode. In customs it has direct consequences: the Harbor Maintenance Fee applies only to cargo arriving by vessel, and an ISF filing is required only for ocean shipments. That makes customs a demanding place to test it, with messy multilingual paperwork, a strict legal definition, and money at stake when the answer is wrong.

The data

We sampled 1,222 unique shipments that brokers completed in Amari’s production system between January and April 2026: 500 ocean, 500 air and 200 truck, plus all 22 completed rail shipments on record.

Our test set has more air and truck shipments than real traffic, so every mode gets enough examples. Real traffic in that period was mostly ocean (93%), then air (6%) and truck (0.7%). The accuracy numbers in this article, like Jev’s 98.9%, are weighted to match that real mix, so they show how a model would do on real traffic. Rail isn’t included because there was no rail traffic in that period; Table 2 shows rail separately. The correct answer for each shipment is the mode on its final record.

The setup

Every model got each shipment exactly as Amari’s production system prepares it: the same document conversion, the same prompt and the same definition of mode of transport. We gave no worked examples, did no fine-tuning and used each model’s default settings.

• Text only. The email plus the text of every attachment, for all four models. Jev accepts text only, and at most 32,000 tokens, so its copy was cut to fit on 50 entries.

• Text + attachments. The same prompt plus the original PDFs, sent the way production sends them. Claude, GPT and Gemini only.

The models were Jev 1.13, Claude Opus 5.5, GPT-6 Astra and Gemini 3.1 Pro. Costs are each vendor’s list price applied to the tokens each call used.

Results

All seven setups landed between 98.9% and 99.4%. What separates them is cost and speed, which span three orders of magnitude.

Figure 1. Accuracy against cost per 1,000 entries. Accuracy is weighted to the production mix of modes; whiskers are 95% bootstrap intervals. The cost axis is logarithmic: each gridline is ten times the one before.

Table 1. All seven setups on the same 1,222 shipments; every setup answered every one. The 95% interval is the range each score could plausibly fall in with a different sample of shipments. The ranges overlap, but a head-to-head check settles it: on the shipments where only one of Jev and a frontier setup was right, the frontier setup was right 16 times and Jev 2–3. That split is too lopsided to be chance, so the frontier models’ lead is real, though small.

Figure 2. Median seconds per shipment. Jev answered 7× faster than the fastest frontier setup.

Per mode, the frontier setups miss almost exactly the same entries: 16 were missed by all six, out of 16 or 17 misses each, and Jev missed 14 of those too. Its extra misses are 16 entries that every frontier setup got right, 12 of them air shipments it called ocean; it was less than 90% sure on 11 of those 16.

Table 2. Right answers by mode, as sampled. 9 of the 22 rail shipments were called ocean by every setup; these are extremely hard edge cases.

A model’s confidence is useful if it tells you which answers to double-check. Suppose a person reviews only the shipments a model was less than 90% sure about. For Jev, that means checking 69 of 1,222 shipments, which would catch 18 of its 30 mistakes. Claude’s confidence works about as well: checking 50 shipments catches 11 of its 17. GPT and Gemini were at least 90% sure about nearly everything, so the same check catches at most 2 of their 16 or 17 mistakes. Jev’s confidence was also the most accurate: on average it was 0.6 points away from how often Jev was actually right, against 1.0 to 2.5 points for the others.

Table 3. What a reviewer would catch by checking only the shipments each setup was less than 90% sure about, out of 1,222. A flagged shipment counts as a positive. Jev’s confidence is its own probability; the LLMs stated theirs in their answers.

What we learned building it

1. An input limit is a deployment decision. Jev accepts 32,000 tokens and rejects anything longer. Its tokenizer counted some dense spreadsheets at more than twice our estimate, so Jev’s copy was cut on 50 entries (4%). Sent uncut, those entries would have been rejected and Jev’s headline accuracy would have been 97.8%. Anyone running it on long shipment files needs the same step.

2. The same PDF is not the same to every API. Claude’s API rejected at least one PDF in 57 shipments (5%) that GPT and Gemini read without complaint, all of them truck entries. Counted as failures, they would have left Claude’s truck score at 142 of 200; re-sending those pages as images fixed every one. Gemini has the opposite constraint: it caps files sent inline at 20 MB, so the largest shipments went through its Files API instead.

3. OCR decides what a text-only model can see. Our first attempt used production’s local OCR fallback, which gives each scanned page five seconds and reads only English. Scanned and Chinese bills of lading came back empty. Switching to the cloud OCR that production uses fixed it, and it matters most for Jev, which never sees the page itself.

4. The hard cases are rare and ambiguous. 9 of 22 rail entries were called ocean by all seven setups. Either the paperwork shows only the ocean leg, or the label is wrong. No model choice fixes that; better documents or a human does.

What this means

For bounded questions like this one, a decision model can take most of the traffic. Jev came within 0.5 points of the frontier models at 74 to 660 times lower cost, answered 7 times faster, and gave the best-calibrated confidence of the seven setups. Its limits are practical: it reads only text, so it depends on OCR, and it rejects inputs over 32,000 tokens.

The frontier models were slightly more accurate, and seeing the original PDFs didn’t help them. Their advantage is the open-ended work this test doesn’t measure: pulling fields out of scanned documents, reasoning across several documents, and explaining a decision.

So use both. Letting Jev answer when it is at least 90% sure and sending the other 6% of shipments to Gemini 3.1 Pro scored 99.3%, against 99.4% for Gemini 3.1 Pro alone, at 13 times lower cost. OpenAI built the same split into ChatGPT, where GPT-5 routes each request to a fast model or a slower reasoning model.

The model also mattered less than the pipeline around it. Swapping models moved accuracy by at most 0.5 points; handling Jev’s input limit moved it by 1.1.