推文推荐用决策模型构建 LLM 评测，并称内部评测显示 Jev 这类 System One 模型很适合做 LLM Judge。作者在 Hamel Husain 关于决策模型评测的文章之外，补充了 Jev-as-a-Judge 智能体评测资源与 arXiv 论文，介绍如何结合 System One 与 System Two 模型。
Learn to build evals, folks.
Highly recommended reading on using decision models for evals.
From our internal evals, System One models like Jev are a great fit for LLM Judges.
If you want to go beyond Hamel's great write-up, I've also put together a few resources on applying decision models to LLM judges.
(bookmark it)
1) Jev-as-a-Judge for Agent Evaluations - https://academy.dair.ai/dashboard/resources/jev-as-a-judge
2) Jev-as-a-Judge: Evaluating Agents with a Decision Model - https://academy.dair.ai/labs/jev-as-a-judge-for-agent-evals
Bonus paper I would recommend, with a few other ideas that combine System One and System Two models: Jev-as-a-Judge paper - https://arxiv.org/abs/2609.26550
New blog post 🎉: How To Use Decision Models For Evals In this post, we show you how to choose a decision model for your task, tune its threshold (don't assume the 0.5 default is good!), and interpret its scores. Lots of cool interactive charts here. https://x.com/i/article/2112552453422104576在 X 查看被引用的帖子
来源：elvis · x.com