怎么 build a eval for LLM （转载） - build eval 的作 - Assbbs
怎么 build a eval for LLM （转载） - build eval 的作用： 用于评估不同 模型的表现（选模型，控制成本等） 作为工具，用于创建满足用户需求的产品 - 比较少见的文章，搬运部分，可以随便看看 来源： 网址 ----------…
怎么 build a eval for LLM （转载） - build eval 的作用： 用于评估不同 模型的表现（选模型，控制成本等） 作为工具，用于创建满足用户需求的产品 - 比较少见的文章，搬运部分，可以随便看看 来源： 网址 ----------------------分割线-------------------------- This is the first write-up in a series about our process of building an " eval " — evaluation — to assess how well AI models perform on prompts related to the SNAP food stamp program. Who is this for? One of the ways all AI models can get better at important domains like safety net benefits is if more domain experts can evaluate the output of models , ideally making those evaluations public. By sharing how we are approaching this for SNAP in some detail — including publishing a SNAP eval — we hope it will make it easier for others to do the same in similar problem spaces that matter for lower income Americans: healthcare e.g. Medicaid , disability benefits, housing, legal help, etc. While evaluation can be a fairly technical topic, we hope these posts reduce barriers to more domain-specific evaluations being created by experts in these high-impact — but complex — areas. What is an "eval"? Roughly, an "eval" is like a test for an AI model. It measures how well a model performs on the dimensions you care about. Here’s a simple example of an eval test case: If I ask an AI model: “What is the maximum SNAP benefit amount for 1 person?” The answer should be $292 What makes evals particularly powerful is that you can automate them . We could write the above as Python code 1 : One of the more interesting aspects of building an eval is that defining what "good" means is usually not as simple as you might think. Generally people think first of accuracy . And accuracy is important in …