Jev是TypeSafe的System One决策模型。给它一个应用状态对象和一个带类型的问题，它会返回一个带类型的回答、一个置信度分数以及每个可能选项的概率（这是Decisions API响应类型，而非聊天端点）。根据TypeSafe的宣传，你可以用像Jev这样的小型判断模型替换前沿聊天模型，从而在分类任务（如本例）中大幅降低成本和延迟。你为此牺牲了多少准确率？

我们将3,080条Banking77客户支持话语分别输入到Jev 1.13和Claude Opus 5中进行测试。Claude Opus 5是截至2026年9月22日，OpenRouter在其排名页面的“任务支出”部分中分类任务花费最多的模型。每条话语都被归类为77种银行意图之一，两个模型都获得了每种意图的一行描述。

下图将Jev和Opus并列展示，显示了它们在准确率、中位延迟以及每千次请求成本方面的对比。

在准确率方面，Jev落后Opus 3.3分，但其在中位数速度上快13倍，成本仅为Opus的1/22。

Jev与Claude Opus 5一览

以下是结果的详细分析。Macro-F1对每个意图赋予相等的权重。

Jev 1.13 Claude Opus 5

准确率 81.0% (79.6至82.3) 84.4% (83.1至85.6)

Macro-F1 80.5% 83.6%

无效响应 0/3,080 0/3,080

延迟 p50 175毫秒 2,266毫秒

延迟 p95 270毫秒 3,004毫秒

延迟 p99 353毫秒 3,835毫秒

计费成本，完整运行 $0.34 $7.44

每1,000次请求的成本 $0.11 $2.42

平均输入令牌数 2,605 3,750 (其中3,725已缓存)

平均输出令牌数 826 17

括号中的数字是95%的自助法置信区间，我们使用3,080个样本计算得出。两个模型均未返回格式错误的响应。所有错误均为标签错误，而非解析失败。

我们的测试方法

通过订阅，您同意接收OpenRouter时事通讯：模型使用数据、产品更新和研究报道，每周约一封电子邮件。您可以通过每封电子邮件中的链接随时取消订阅。请参阅我们的隐私政策。

Banking77是来自PolyAI的语料级别客户支持意图数据集，采用CC BY 4.0许可证。这些话语很短，中位长度为9个单词，每个话语都标记为77种意图之一。当我们在Banking77上运行Jev和Opus时，我们使用了完整的测试集，包含3,080个样本，每种意图40个。有些意图非常接近，模型不能仅仅依靠几个关键词就停止阅读。例如card_arrival与card_delivery_estimate的区别。

我们仅根据标签名称为每个意图写了一行标准，完全没有查看测试数据，并将完全相同的标准列表提供给Jev和Opus。为了评估Jev，我们将每条话语作为Decisions API Choice问题发送，将77条标准作为选项。对于Opus，我们构建了一个提示，其中包含列出所有标准的系统消息，以及仅包含话语的用户消息。我们以零温度运行Opus，关闭推理功能，并使用严格的JSON模式响应格式，其中包含77个标签的枚举。由于系统消息在每个请求中都是相同的，因此我们开启了提示缓存。两个模型都在同一台机器上运行，我们每次发送8个并发请求，并将延迟测量为通过OpenRouter观察到的客户端往返时间。

Jev有多准确？

让我们谈谈数字。在Banking77上，Jev的准确率为81.0%，Opus为84.4%。配对自助法给出的95%置信区间为2.3到4.4分，因此Opus领先约3分不太可能是噪声。

从好的方面来看，这两个模型有大量的共识。它们在89.3%的话语上达成一致。在它们意见分歧的地方，Opus单独正确识别了175个，Jev正确识别了72个。

从坏的方面来看，两者都远低于那些在所有10,003个Banking77训练样本上进行微调的编码器所报告的90年代初的水平。这就是依赖一行标准而不是进行完整微调的成本。

就各类别而言，Opus 在 77 个意图中的 35 个上领先于 Jev，Jev 在 15 个上领先，其余 27 个打平。Opus 优势最大的是 receiving_money（接收金钱），其准确率为 80.0%，而 Jev 为 52.5%。与此同时，Jev 在 compromised_card（被盗银行卡）类别表现亮眼，准确率达到 95.0%，而 Opus 仅为 70.0%。Opus 倾向于将盗用的卡信息误判为未识别的支付。

Jev 的速度有多快？成本有多低？

非常快。Jev 的中位往返时间为 175 毫秒，p95（第 95 百分位数）为 270 毫秒。相比之下，Opus 在不启用推理模式的情况下，中位往返时间为 2,266 毫秒，p95 为 3,004 毫秒。这意味着 Jev 最慢的一次调用（约 1.6 秒）也比 Opus 最快的一次调用（约 1.9 秒）还要快。

Jev 的总费用为 0.34 美元，即每千次请求 0.11 美元。Opus 的总费用为 7.44 美元，即每千次请求 2.42 美元。上述 Opus 的数据已缓存了 3,700 token 的系统提示词，因此所有请求均按缓存读取费率计费，而非列表费率。若未使用缓存，Opus 的列表费率约为每千次请求 19 美元。因此，如果你在使用前沿模型配合标签分类法，请务必缓存系统提示词。

基于 Jev 置信度的路由

Jev 提供置信度分数，但这并非经过校准的概率。在此测试中，它在中等置信度区间高估了自身的准确性。尽管如此，其排序效果依然良好。在置信度 ≥0.99 的 58% 的语句中，准确率为 96.3%。在置信度 <0.5 的 3.5% 的语句中，准确率为 29.6%。

这种排序足以实现级联处理。设定一个阈值，高于该阈值的请求由 Jev 处理，其余请求转发给 Opus。以下是不同阈值下的准确率及每千次请求的成本，范围从仅使用 Jev 到仅使用 Opus：

在阈值 0.90 时，76% 的流量绕过了 Opus。作为交换，与仅使用 Opus 相比，你的准确率最多降低 0.4 个百分点，而成本则降低了 3.5 倍。Opus 正确识别了 Jev 遗漏的 175 条语句。这些案例的中位置信度为 0.67，其中 85% 低于 0.90。因此，Jev 通常能知道自己何时只是在猜测。

注意事项

让我们狭义地谈谈我们实际进行的测试：一个数据集、一个领域、一种提示词设计，以及某个下午的一个十五分钟窗口。

如果此处存在 Opus 的记忆优势，由于 Banking77 数据集发布于 2020 年，其得分可能因此略被高估。但仅凭此次运行无法断定这一点。

另一个要点是，两个模型都在一个类别上失手，因为判定标准仅基于标签名称编写，未查看示例消息。在此案例中，其中一个标签是 get_physical_card（获取实体卡），涵盖关于 PIN 码是否单独发送的查询。仅从名称几乎不可能猜出这一含义。在该类别上，每个模型的得分均为 0/40，并将大部分消息路由到了 change_pin（更改 PIN 码）。若排除该类别，Jev 的准确率为 82.1%，Opus 为 85.5%。但物流团队会使用预留的训练数据而非测试集来迭代优化判定标准，这样两个模型的得分都会提高。

我们展示的级联表使用了与准确率测量完全相同的 3,080 个示例，因此这是一个上限值。请根据你的实际流量选择阈值。

最后，这些测试是在 Opus 处于“推理关闭”模式且启用结构化输出时运行的。Opus 未在“推理开启”模式下进行测试，也未使用其他模式或少样本示例进行测试。

这意味着什么

如果准确率的三个百分点优势超过每千次请求 2.42 美元的成本差异，请选择 Opus。如果你处理的是高流量、低延迟场景，或需要备用槽位，仅使用 Jev 的准确率已接近 Opus 3.3 个百分点。更优的是，基于 Jev 的置信度分数运行级联处理，使我们的准确率差距缩小至 0.4 个百分点，而成本仅为原来的不到 30%。

Decisions API 参考文档详细介绍了基本请求的结构。Jev 食谱（cookbook）逐步指导你在 TypeScript 中实现一个可工作的“选择”问题。经过 Jev 验证的级联食谱展示了代码中的升级模式：由廉价模型起草，Jev 检查草稿，并由前沿模型仅处理未通过检查的部分。Banking77 测试集是 PolyAI 存储库中一个包含 3,080 行的 CSV 文件。

要在你自己的积压任务上运行此类标注工作，Classify and Tag Text at Scale with Jev 食谱涵盖了在速率限制内进行批处理、从已标注样本中为每个标签选择阈值，以及计算每 1,000 个项目的成本。如果你刚接触 Jev，请先阅读“什么是 Jev？”，然后参考“Jev vs LLM”以了解何时用决策模型替代生成式调用。Jev 文档中心列出了 OpenRouter 上的所有 Jev 指南和食谱。

常见问题

与 Claude Opus 5 相比，Jev 在意图分类方面的准确率如何？

在 2026 年 9 月 22 日运行的、包含 77 个意图共 3,080 条话语的 Banking77 测试集上，Claude Opus 5 达到了 84.4% 的准确率和 83.6% 的宏观 F1 分数，而 Jev 1.13 达到了 81.0% 的准确率和 80.5% 的宏观 F1 分数。准确率上的配对差距为 3.3 个百分点，95% 的自助法置信区间为 2.3 到 4.4 个百分点。两个模型在 89.3% 的话语上达成一致。

Jev 在分类任务中比前沿模型快多少？

在客户端观察到的中位数和 p95（第 95 百分位数）往返延迟方面，Jev 的中位数为 175 毫秒，p95 为 270 毫秒；而禁用推理功能的 Claude Opus 5 的中位数为 2,266 毫秒，p95 为 3,004 毫秒，中位数高出约 13 倍。这些数字是在同一客户端下、8 个并发请求的环境中测量的。它们包括网络时间以及提供商系统中实际推理发生前的任何排队时间。

在 OpenRouter 上使用 Jev 对文本进行分类的成本是多少？

Banking77 的所有 3,080 条请求在 typesafe/jev-1.13 上的总费用为 0.34 美元，即每千次请求 0.11 美元，或每次请求约 0.0001 美元。在使用针对 77 标签系统提示词的提示缓存的情况下，相同请求在 anthropic/claude-opus-5 上的总费用为 7.44 美元，即每千次请求 2.42 美元。如果不使用缓存，Opus 的列表价格约为每千次请求 19 美元。

我可以使用 Jev 的置信度分数来决定何时回退到更大的模型吗？

可以，在我们的数据中，它确实可作为排序信号。我们测量得出，在置信度至少为 0.99 的 58% 的话语中，Jev 的准确率为 96.3%；而在置信度低于 0.5 的 3.5% 的话语中，准确率为 29.6%。将所有置信度低于 0.9 的请求路由到 Claude Opus 5，可将准确率恢复至 84.0%，仅比单独使用 Opus 低 0.4 个百分点，且成本为每千次请求 0.69 美元。由于该分数并未校准为概率值，请根据你的自有数据选择合适的阈值。

Jev is TypeSafe’s System One decision model. Give it an app state object and a typed question, and it gives you back a typed answer, a confidence score, and the probability for each possible option (the Decisions API response type, not the chat endpoints). According to the TypeSafe pitch, you can replace a frontier chat model with a small judgment model like Jev and get significant cuts in both cost and latency, on classification tasks such as this one. How much accuracy are you sacrificing?

We ran 3,080 Banking77 customer support utterances through both Jev 1.13 and Claude Opus 5 , the model OpenRouter users spent the most on for classification in the Task spend section of its rankings page as of 22 September 2026. Each utterance gets sorted into one of 77 banking intents, and both models got the same one-line description of each intent.

The chart below puts Jev and Opus side by side, showing how they compare on accuracy, median latency, and cost per thousand requests.

Jev comes in 3.3 points behind Opus on accuracy, but it’s 13 times faster at the median and costs just 1/22 of what Opus does.

Jev vs Claude Opus 5 at a glance

Here’s a closer look at the results. Macro-F1 gives equal weight to every intent.

Jev 1.13 Claude Opus 5

Accuracy 81.0% (79.6 to 82.3) 84.4% (83.1 to 85.6)

Macro-F1 80.5% 83.6%

Invalid responses 0 of 3,080 0 of 3,080

Latency p50 175 ms 2,266 ms

Latency p95 270 ms 3,004 ms

Latency p99 353 ms 3,835 ms

Billed cost, full run $0.34 $7.44

Cost per 1,000 requests $0.11 $2.42

Mean input tokens 2,605 3,750 (3,725 cached)

Mean output tokens 826 17

The numbers in parentheses are 95% bootstrap confidence intervals, which we calculated using 3,080 examples. Neither model returned a malformed response. Every error was a wrong label, not a parse failure.

How we ran it

By subscribing you agree to receive the OpenRouter newsletter: model usage data, product updates, and research reports, about one email a week. Unsubscribe anytime via the link in every email. See our Privacy Policy .

Banking77 is an utterance-level customer support intent dataset from PolyAI with the CC BY 4.0 license. The utterances are short, median nine words, and each one is labeled with one of 77 intents. When we ran Jev and Opus on Banking77, we used the full test split, which has 3,080 examples, 40 per intent. Some of the intents are close enough that the models can’t just pick up on a handful of keywords and stop reading. Think card_arrival vs. card_delivery_estimate .

We wrote up a one-line criterion for each intent from just the label name, without looking at the test data at all, and we gave the same exact list of criteria to both Jev and Opus. To score Jev, we sent each utterance as a Decisions API Choice question with the 77 criteria as the options. For Opus, we built a prompt where it saw a system message listing all the criteria, and then a user message with just the utterance. We ran Opus at temperature zero, with reasoning turned off, and using a strict JSON schema response format with an enum of the 77 labels. We turned on prompt caching for the system message, since it was identical on every request. Both models ran from the same machine, and we sent 8 concurrent requests at a time and measured latency as the client-observed round-trip through OpenRouter.

How accurate is Jev?

Let’s talk numbers. On Banking77, Jev is at 81.0%, Opus at 84.4%. A paired bootstrap gives a 95% CI of 2.3 to 4.4 points, so Opus ahead by about three points is unlikely to be noise.

On the upside, these two have a ton of agreement. They agree on 89.3% of the utterances. Where they split, Opus gets 175 right on its own, Jev gets 72.

On the downside, both are still sitting well below the low 90s that encoders fine-tuned on all 10,003 Banking77 training examples report. That’s the cost of relying on one-line criteria instead of doing a full fine-tune.

By class, Opus leads Jev on 35 of the 77 intents, Jev on 15, tied on 27. Opus’s biggest lead is on receiving_money , where it’s at 80.0% compared to Jev’s 52.5%. Meanwhile, Jev shines on compromised_card where it’s at 95.0% while Opus is at 70.0%. Opus tended to mistake stolen card details for an unrecognized payment.

How fast and how cheap is Jev?

Very quickly, Jev’s median round trip was 175 ms and p95 was 270 ms. Opus, on the other hand, without reasoning mode, had a median round trip of 2,266 ms and p95 of 3,004 ms. That means the slowest Jev call (~1.6s) was faster than the fastest Opus call (~1.9s).

The total for Jev was $0.34, or $0.11 per thousand requests. Opus came in at $7.44, or $2.42 per thousand requests. That Opus figure has the 3,700-token system prompt cached, so all the requests were at the cache-read rate instead of the list rate. Without caching, the list rate for Opus would have been about $19 per thousand requests. So if you’re using a label taxonomy with a frontier model, cache that system prompt.

Routing on Jev’s confidence

Jev gives you a confidence score, but it’s not a calibrated probability. Here, it overestimated its own accuracy mid-range. Still, it ranks well. For 58% of utterances with confidence ≥0.99, accuracy is 96.3%. For the 3.5% below 0.5, accuracy is 29.6%.

That ordering is all you need to cascade . Accept Jev above a given threshold, send the rest to Opus. Here’s accuracy and cost per thousand at each threshold, ranging from Jev only down to Opus only.

Threshold Handled by Jev Accuracy Cost per 1,000

Jev only 100% 81.0% $0.11

0.99 57.8% 84.3% $1.13

0.95 68.1% 84.2% $0.88

0.90 75.9% 84.0% $0.69

0.80 82.7% 83.6% $0.53

0.70 87.3% 83.2% $0.42

Opus only 0% 84.4% $2.42

At threshold 0.90, 76% of traffic bypasses Opus. In exchange, your accuracy is reduced by at most 0.4 points relative to Opus alone, and your cost cut by a factor of 3.5. Opus got right 175 utterances that Jev missed. The median confidence of those cases was 0.67, and 85% were below 0.90. So Jev usually knows when it’s just guessing.

Caveats

So let’s talk narrowly about the actual test we did: one dataset, one domain, one prompt design, and one fifteen-minute window on one afternoon.

If there is an Opus memorization edge here, its score may be slightly flattered by it, because Banking77 was published in 2020. But you can’t tell from this run alone.

The other point is that both models stumbled on one class, because the criteria were written from label names only, without seeing example messages. In this case, one of the labels was get_physical_card , which covers queries about whether the PIN is delivered separately. It’s practically impossible to guess that from the name. Each model scored 0 out of 40 on that class, routing most messages to change_pin instead. Excluding that class alone, Jev hit 82.1% and Opus 85.5%. But a shipping team would iterate on the criteria using held-out training data, not the test set, and both models would get better.

The cascade table we showed used exactly the same 3,080 examples used for the accuracy measurement, so this is an upper bound. Choose your threshold based on your own traffic.

And lastly, these tests were run with Opus in “reasoning off” mode and structured outputs on. Opus was not tested in “reasoning on” mode, or with other schemas or few-shot examples.

What this means

If three points in accuracy outweigh $2.42 per thousand requests, go with Opus. If you’re dealing with high volume, low latency, or need a fallback slot, Jev alone got within 3.3 points of Opus. Even better, running a cascade on Jev’s confidence score brought us within 0.4 points of Opus, for under 30% of the cost.

The Decisions API reference walks through the basic request shape. The Jev cookbook walks you through a working Choice question in TypeScript. The Jev-verified cascade cookbook has the escalation pattern in code, with a cheap model drafting, Jev checking the draft, and a frontier model handling only what fails the check. The Banking77 test split is a 3,080-line CSV in the PolyAI repository .

To run a labeling job like this one over your own backlog, the Classify and Tag Text at Scale with Jev cookbook covers batching within rate limits, picking a threshold per label from a labeled sample, and computing cost per 1,000 items. If Jev is new to you, start with What Is Jev? , then Jev vs LLM for when a decision model replaces a generative call. The Jev documentation hub lists every Jev guide and cookbook on OpenRouter.

Frequently Asked Questions

How accurate is Jev compared to Claude Opus 5 on intent classification?

On the Banking77 test split consisting of 3,080 utterances across 77 intents run on 22 September 2026, Claude Opus 5 achieved 84.4% accuracy and 83.6% macro-F1, while Jev 1.13 achieved 81.0% accuracy and 80.5% macro-F1. The paired gap was 3.3 percentage points on accuracy with a 95% bootstrap interval from 2.3 to 4.4 percentage points. The models agreed on 89.3% of utterances.

How much faster is Jev than a frontier model for classification?

On median and p95 client-observed round trip latency, Jev achieved median 175 ms and p95 270 ms, while Claude Opus 5 with reasoning disabled achieved median 2,266 ms and p95 3,004 ms, about 13 times higher at the median. These numbers are measured from the same client under 8 concurrent requests. They include network time and any queueing in the provider’s system before the actual inference.

How much does it cost to classify text with Jev on OpenRouter?

All 3,080 of the Banking77 requests were billed $0.34 total on typesafe/jev-1.13, which is $0.11 per thousand requests or about $0.0001 per request. Using prompt caching on the 77-label system prompt, the same requests billed $7.44 total on anthropic/claude-opus-5, or $2.42 per thousand. Without caching, Opus would have billed about $19 per thousand at list price.

Can I use Jev’s confidence score to decide when to fall back to a bigger model?

Yes, and in our data it works as a ranking signal. We measured Jev as 96.3% accurate on the 58% of utterances with confidence at least 0.99, and 29.6% accurate on the 3.5% of utterances with confidence under 0.5. Routing everything under 0.9 confidence to Claude Opus 5 recovered 84.0% accuracy, within 0.4 points of Opus alone, at $0.69 per thousand requests. Because the score is not calibrated as a probability, pick the threshold that works for your own data.