尧图网站建设 
Pairwise Comparison Evaluation【免费下载链接】Agent-Skills-for-Context-EngineeringA comprehensive collection of Agent Skills for context engineering, multi-agent architectures, and production agent systems. Use when building, optimizing, or debugging agent systems that require effective context management.项目地址: https://gitcode.com/GitHub_Trending/ag/Agent-Skills-for-Context-EngineeringYou are an expert evaluator comparing two AI-generated responses to the same prompt.Your TaskCompare Response A and Response B, then determine which better satisfies the requirements. You must:Analyze each response independently firstCompare them directly on each criterionMake a final determination with confidence levelImportant GuidelinesEvaluate content quality, not superficial differencesDo NOT prefer responses simply because they are longerDo NOT prefer responses based on their position (A vs B)Focus on the specified criteriaTies are acceptable when responses are genuinely equivalentExplain your reasoning before stating the winnerOriginal Prompt/Task{{original_prompt}}{{#if context}}Additional Context{{context}} {{/if}}Response Aresponse_a {{response_a}} /response_aResponse Bresponse_b {{response_b}} /response_bComparison Criteria{{#each criteria}}{{this}}{{/each}}Your EvaluationStep 1: Independent AnalysisFirst, briefly analyze each response:Response A Analysis:Key strengths:Key weaknesses:Notable features:Response B Analysis:Key strengths:Key weaknesses:Notable features:Step 2: Head-to-Head ComparisonFor each criterion, compare the responses:{{#each criteria}}{{this}}:Response A: [assessment]Response B: [assessment]Winner for this criterion: [A / B / TIE] {{/each}}Step 3: Final DeterminationBased on your analysis:Winner: [A / B / TIE]Confidence: [0.0-1.0]Reasoning: [Why this response is better overall]Key Differentiators: [What most strongly distinguishes the winner]Format your response as structured JSON:{ analysis: { responseA: { strengths: [..., ...], weaknesses: [..., ...] }, responseB: { strengths: [..., ...], weaknesses: [..., ...] } }, comparison: [ { criterion: {{criterion}}, aAssessment: ..., bAssessment: ..., winner: A | B | TIE, reasoning: ... } ], result: { winner: A | B | TIE, confidence: 0.85, reasoning: ..., differentiators: [..., ...] } }这段模板的工程价值在于它把评估方法论显式编码进了提示词角色定义、任务步骤、防偏差红线、输入容器XML 式标签包裹、输出契约结构化 JSON五层结构一应俱全与 [prompts/index.md](https://link.gitcode.com/i/5d1718b5f2d3d8111d81677382665fa3) 中总结的清晰角色定义、显式指令、结构化输出、护栏四大设计原则一一对应。 ## 变量体系五个输入占位符的职责与必需性 模板通过五个变量完成参数化变量表如下与文档一致并补充说明 | 变量 | 描述 | 必需 | |------|------|------| | original_prompt | 两个响应共同回答的原始提示词 | 是 | | context | 附加上下文如 RAG 文档、历史 | 否 | | response_a | 第一个待比较响应 | 是 | | response_b | 第二个待比较响应 | 是 | | criteria | 比较维度列表 | 是 | - **original_prompt** 是评估的锚点评判必须回到任务本身防止脱离任务空谈响应优劣 - **context** 为可选条件段由 {{#if context}}...{{/if}} 包裹仅在传入时渲染。这一点在工程实现中同样被保留——见下文 evaluatePair 中的 context ? ... : 条件拼接 - **criteria** 使用 {{#each criteria}} 迭代渲染每个标准都会出现在Comparison Criteria清单与 Step 2 的逐标准对比中数量不限但建议控制在可判定的合理范围。 在 [源码实现](https://link.gitcode.com/i/2fbf94e83e2a96d366b1d2ada591a9a1) 中这五个变量被映射为类型安全的 Zod 输入 SchemaresponseA、responseB、prompt 为必填 stringcontext 为 optional stringcriteria 为至少含一个元素的 z.array(z.string()).min(1)从运行时层面保证模板所需变量必然齐备。 ## 三步评估流程独立分析 → 逐标准对比 → 最终判定 模板将评判过程强制拆解为三个串行步骤这是它区别于直接让模型给出结论式提示词的关键 **Step 1独立分析Independent Analysis**。先分别分析 Response A 与 Response B列出各自的关键优势Key strengths、关键弱点Key weaknesses与显著特征Notable features。这一步骤的核心价值是阻断先入为主——在对比发生之前每个响应都被单独审视一遍避免模型带着对另一方的成见草率下结论。 **Step 2逐标准对比Head-to-Head Comparison**。对 criteria 中的每个维度单独比较分别给出 A、B 两方的评估结论并给出该维度上的胜者A / B / TIE。逐标准分解避免了整体感觉对评判的污染也让最终结论可回溯——每个维度的裁决都有独立依据。 **Step 3最终判定Final Determination**。综合以上分析输出 WinnerA / B / TIE、Confidence0.0-1.0 的置信度、Reasoning为什么整体上该响应更优与 Key Differentiators最能区分胜负的关键差异点。 三步结构与 [源码中 evaluatePair 的 prompt 组装](https://link.gitcode.com/i/7c961e5cc510e9867a5326e3e8c5f436) 完全对应——源码同样要求First analyze each response independently, then compare them并要求返回包含 analysis、comparison、result 三段的 JSON。README 的测试记录也印证了这一点强制要求理由justification能显著提升评估可靠性测试中所有打分均包含大于 20 字符、附带具体证据的说明。 ## 位置偏差消除双轮交换评估算法 模板在 Important Guidelines 中明确写了 Do NOT prefer responses based on their position (A vs B)但仅靠提示词约束远不足以根除位置偏差。文档因此给出了生产环境的关键配套方案**位置交换position swapping**——对同一对响应以正序、倒序各评估一次再对结果做交叉映射与一致性检查。 文档提供的核心算法如下保留完整代码可直接落地 typescript async function compareWithPositionSwap(a: string, b: string, criteria: string[]) { // First evaluation: A first, B second const eval1 await evaluate({ response_a: a, response_b: b, criteria }); // Second evaluation: B first, A second const eval2 await evaluate({ response_a: b, response_b: a, criteria }); // Map eval2 result back (swap winner) const eval2Winner eval2.winner A ? B : eval2.winner B ? A : TIE; // Check consistency if (eval1.winner eval2Winner) { return { winner: eval1.winner, confidence: (eval1.confidence eval2.confidence) / 2, consistent: true }; } else { // Inconsistent - likely close, return TIE or lower confidence return { winner: TIE, confidence: 0.5, consistent: false, note: Evaluation inconsistent across positions }; } }算法要点两轮评估第一轮 A 在前 B 在后第二轮 B 在前 A 在后结果映射第二轮若判定 A 赢映射回真实语义即原 A 赢第二轮判定 B 赢映射回原 B 赢一致性裁决若两轮映射后胜者一致取平均置信度作为最终置信度若不一致说明两条响应实力接近、判定受位置干扰此时保守地返回TIE并将置信度下调至 0.5。这套算法在 pairwise-compare.ts 的executePairwiseCompare中被完整实现并且做了更细粒度的增强不仅对最终胜者做一致性检查还对每个 criterion 的逐维度胜者做交叉映射比对不一致的维度裁决降级为 TIE最终把非 TIE 的维度整理为differentiators列表。工具文档 pairwise-compare.md 还给出了生产建议位置交换评估默认开启swapPositions默认true。结构化输出契约从提示词 JSON 到 Zod 校验模板的 Step 3 末尾要求Format your response as structured JSON这份输出契约是提示词工程中结构化输出Structured Output原则的直接体现见 prompts/index.md 的设计原则第 3 条。提示词要求的 JSON 包含三层analysisA/B 双方的优势与弱点、comparison逐标准的评估、裁决与理由、result最终胜者、置信度、整体理由与关键区分点。在工程侧PairwiseCompareOutputSchema用 Zod 对模型输出做了更完整的契约约束winner严格限定为A | B | TIE枚举confidence限定在[0, 1]区间comparison为对象数组每个对象必须含criterion、winner、aAssessment、bAssessment、reasoning五个字段额外包含positionConsistency两轮评估胜者与一致性标志与metadata评估耗时、所用模型、是否交换位置。代码中先JSON.parse(result.text)解析模型输出再由 Zod Schema 进行运行时校验防止脏数据流入下游。注意 README 中描述的positionConsistency字段为可选仅swapPositions: true时返回这与单轮评估模式无交换的输出形态相互兼容。实战示例一次完整的成对比较调用文档给出了一个可直接运行的输入样例主题为解释规律运动的好处{ original_prompt: Explain the benefits of regular exercise, response_a: Regular exercise offers numerous benefits including improved cardiovascular health, stronger muscles, better mental health, and increased energy levels. Studies show that even 30 minutes of moderate exercise daily can significantly reduce the risk of heart disease., response_b: Working out is great for you. It helps your heart, makes you stronger, and improves your mood. You should try to exercise most days of the week., criteria: [accuracy, specificity, actionability, engagement] }可见 Response A 信息密度高、有具体数据支撑Response B 笼统空泛——两者质量差异明显是验证评估器敏感度的理想样例。在工程中调用该评估则通过EvaluatorAgent完成。相关 可运行示例 展示了标准调用姿势const result await agent.compare({ responseA, responseB, prompt: Explain the benefits of regular exercise, criteria: [accuracy, completeness, actionability, clarity], allowTie: true, swapPositions: true // Mitigate position bias }); if (result.success) { console.log(Winner: Response ${result.winner}); console.log(Confidence: ${(result.confidence * 100).toFixed(0)}%); console.log(Position Consistency: ${result.positionConsistency.consistent ? Yes : No}); }【免费下载链接】Agent-Skills-for-Context-EngineeringA comprehensive collection of Agent Skills for context engineering, multi-agent architectures, and production agent systems. Use when building, optimizing, or debugging agent systems that require effective context management.项目地址: https://gitcode.com/GitHub_Trending/ag/Agent-Skills-for-Context-Engineering创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考 Pairwise Comparison Evaluation
Pairwise Comparison Evaluation【免费下载链接】Agent-Skills-for-Context-EngineeringA comprehensive collection of Agent Skills for context engineering, multi-agent architectures, and production agent systems. Use when building, optimizing, or debugging agent systems that require effective context management.项目地址: https://gitcode.com/GitHub_Trending/ag/Agent-Skills-for-Context-EngineeringYou are an expert evaluator comparing two AI-generated responses to the same prompt.Your TaskCompare Response A and Response B, then determine which better satisfies the requirements. You must:Analyze each response independently firstCompare them directly on each criterionMake a final determination with confidence levelImportant GuidelinesEvaluate content quality, not superficial differencesDo NOT prefer responses simply because they are longerDo NOT prefer responses based on their position (A vs B)Focus on the specified criteriaTies are acceptable when responses are genuinely equivalentExplain your reasoning before stating the winnerOriginal Prompt/Task{{original_prompt}}{{#if context}}Additional Context{{context}} {{/if}}Response Aresponse_a {{response_a}} /response_aResponse Bresponse_b {{response_b}} /response_bComparison Criteria{{#each criteria}}{{this}}{{/each}}Your EvaluationStep 1: Independent AnalysisFirst, briefly analyze each response:Response A Analysis:Key strengths:Key weaknesses:Notable features:Response B Analysis:Key strengths:Key weaknesses:Notable features:Step 2: Head-to-Head ComparisonFor each criterion, compare the responses:{{#each criteria}}{{this}}:Response A: [assessment]Response B: [assessment]Winner for this criterion: [A / B / TIE] {{/each}}Step 3: Final DeterminationBased on your analysis:Winner: [A / B / TIE]Confidence: [0.0-1.0]Reasoning: [Why this response is better overall]Key Differentiators: [What most strongly distinguishes the winner]Format your response as structured JSON:{ analysis: { responseA: { strengths: [..., ...], weaknesses: [..., ...] }, responseB: { strengths: [..., ...], weaknesses: [..., ...] } }, comparison: [ { criterion: {{criterion}}, aAssessment: ..., bAssessment: ..., winner: A | B | TIE, reasoning: ... } ], result: { winner: A | B | TIE, confidence: 0.85, reasoning: ..., differentiators: [..., ...] } }这段模板的工程价值在于它把评估方法论显式编码进了提示词角色定义、任务步骤、防偏差红线、输入容器XML 式标签包裹、输出契约结构化 JSON五层结构一应俱全与 [prompts/index.md](https://link.gitcode.com/i/5d1718b5f2d3d8111d81677382665fa3) 中总结的清晰角色定义、显式指令、结构化输出、护栏四大设计原则一一对应。 ## 变量体系五个输入占位符的职责与必需性 模板通过五个变量完成参数化变量表如下与文档一致并补充说明 | 变量 | 描述 | 必需 | |------|------|------| | original_prompt | 两个响应共同回答的原始提示词 | 是 | | context | 附加上下文如 RAG 文档、历史 | 否 | | response_a | 第一个待比较响应 | 是 | | response_b | 第二个待比较响应 | 是 | | criteria | 比较维度列表 | 是 | - **original_prompt** 是评估的锚点评判必须回到任务本身防止脱离任务空谈响应优劣 - **context** 为可选条件段由 {{#if context}}...{{/if}} 包裹仅在传入时渲染。这一点在工程实现中同样被保留——见下文 evaluatePair 中的 context ? ... : 条件拼接 - **criteria** 使用 {{#each criteria}} 迭代渲染每个标准都会出现在Comparison Criteria清单与 Step 2 的逐标准对比中数量不限但建议控制在可判定的合理范围。 在 [源码实现](https://link.gitcode.com/i/2fbf94e83e2a96d366b1d2ada591a9a1) 中这五个变量被映射为类型安全的 Zod 输入 SchemaresponseA、responseB、prompt 为必填 stringcontext 为 optional stringcriteria 为至少含一个元素的 z.array(z.string()).min(1)从运行时层面保证模板所需变量必然齐备。 ## 三步评估流程独立分析 → 逐标准对比 → 最终判定 模板将评判过程强制拆解为三个串行步骤这是它区别于直接让模型给出结论式提示词的关键 **Step 1独立分析Independent Analysis**。先分别分析 Response A 与 Response B列出各自的关键优势Key strengths、关键弱点Key weaknesses与显著特征Notable features。这一步骤的核心价值是阻断先入为主——在对比发生之前每个响应都被单独审视一遍避免模型带着对另一方的成见草率下结论。 **Step 2逐标准对比Head-to-Head Comparison**。对 criteria 中的每个维度单独比较分别给出 A、B 两方的评估结论并给出该维度上的胜者A / B / TIE。逐标准分解避免了整体感觉对评判的污染也让最终结论可回溯——每个维度的裁决都有独立依据。 **Step 3最终判定Final Determination**。综合以上分析输出 WinnerA / B / TIE、Confidence0.0-1.0 的置信度、Reasoning为什么整体上该响应更优与 Key Differentiators最能区分胜负的关键差异点。 三步结构与 [源码中 evaluatePair 的 prompt 组装](https://link.gitcode.com/i/7c961e5cc510e9867a5326e3e8c5f436) 完全对应——源码同样要求First analyze each response independently, then compare them并要求返回包含 analysis、comparison、result 三段的 JSON。README 的测试记录也印证了这一点强制要求理由justification能显著提升评估可靠性测试中所有打分均包含大于 20 字符、附带具体证据的说明。 ## 位置偏差消除双轮交换评估算法 模板在 Important Guidelines 中明确写了 Do NOT prefer responses based on their position (A vs B)但仅靠提示词约束远不足以根除位置偏差。文档因此给出了生产环境的关键配套方案**位置交换position swapping**——对同一对响应以正序、倒序各评估一次再对结果做交叉映射与一致性检查。 文档提供的核心算法如下保留完整代码可直接落地 typescript async function compareWithPositionSwap(a: string, b: string, criteria: string[]) { // First evaluation: A first, B second const eval1 await evaluate({ response_a: a, response_b: b, criteria }); // Second evaluation: B first, A second const eval2 await evaluate({ response_a: b, response_b: a, criteria }); // Map eval2 result back (swap winner) const eval2Winner eval2.winner A ? B : eval2.winner B ? A : TIE; // Check consistency if (eval1.winner eval2Winner) { return { winner: eval1.winner, confidence: (eval1.confidence eval2.confidence) / 2, consistent: true }; } else { // Inconsistent - likely close, return TIE or lower confidence return { winner: TIE, confidence: 0.5, consistent: false, note: Evaluation inconsistent across positions }; } }算法要点两轮评估第一轮 A 在前 B 在后第二轮 B 在前 A 在后结果映射第二轮若判定 A 赢映射回真实语义即原 A 赢第二轮判定 B 赢映射回原 B 赢一致性裁决若两轮映射后胜者一致取平均置信度作为最终置信度若不一致说明两条响应实力接近、判定受位置干扰此时保守地返回TIE并将置信度下调至 0.5。这套算法在 pairwise-compare.ts 的executePairwiseCompare中被完整实现并且做了更细粒度的增强不仅对最终胜者做一致性检查还对每个 criterion 的逐维度胜者做交叉映射比对不一致的维度裁决降级为 TIE最终把非 TIE 的维度整理为differentiators列表。工具文档 pairwise-compare.md 还给出了生产建议位置交换评估默认开启swapPositions默认true。结构化输出契约从提示词 JSON 到 Zod 校验模板的 Step 3 末尾要求Format your response as structured JSON这份输出契约是提示词工程中结构化输出Structured Output原则的直接体现见 prompts/index.md 的设计原则第 3 条。提示词要求的 JSON 包含三层analysisA/B 双方的优势与弱点、comparison逐标准的评估、裁决与理由、result最终胜者、置信度、整体理由与关键区分点。在工程侧PairwiseCompareOutputSchema用 Zod 对模型输出做了更完整的契约约束winner严格限定为A | B | TIE枚举confidence限定在[0, 1]区间comparison为对象数组每个对象必须含criterion、winner、aAssessment、bAssessment、reasoning五个字段额外包含positionConsistency两轮评估胜者与一致性标志与metadata评估耗时、所用模型、是否交换位置。代码中先JSON.parse(result.text)解析模型输出再由 Zod Schema 进行运行时校验防止脏数据流入下游。注意 README 中描述的positionConsistency字段为可选仅swapPositions: true时返回这与单轮评估模式无交换的输出形态相互兼容。实战示例一次完整的成对比较调用文档给出了一个可直接运行的输入样例主题为解释规律运动的好处{ original_prompt: Explain the benefits of regular exercise, response_a: Regular exercise offers numerous benefits including improved cardiovascular health, stronger muscles, better mental health, and increased energy levels. Studies show that even 30 minutes of moderate exercise daily can significantly reduce the risk of heart disease., response_b: Working out is great for you. It helps your heart, makes you stronger, and improves your mood. You should try to exercise most days of the week., criteria: [accuracy, specificity, actionability, engagement] }可见 Response A 信息密度高、有具体数据支撑Response B 笼统空泛——两者质量差异明显是验证评估器敏感度的理想样例。在工程中调用该评估则通过EvaluatorAgent完成。相关 可运行示例 展示了标准调用姿势const result await agent.compare({ responseA, responseB, prompt: Explain the benefits of regular exercise, criteria: [accuracy, completeness, actionability, clarity], allowTie: true, swapPositions: true // Mitigate position bias }); if (result.success) { console.log(Winner: Response ${result.winner}); console.log(Confidence: ${(result.confidence * 100).toFixed(0)}%); console.log(Position Consistency: ${result.positionConsistency.consistent ? Yes : No}); }【免费下载链接】Agent-Skills-for-Context-EngineeringA comprehensive collection of Agent Skills for context engineering, multi-agent architectures, and production agent systems. Use when building, optimizing, or debugging agent systems that require effective context management.项目地址: https://gitcode.com/GitHub_Trending/ag/Agent-Skills-for-Context-Engineering创作声明：本文部分内容由AI辅助生成（AIGC），仅供参考