验收 · 怎么收 agent 交回的活Review · How to Collect What the Agent Hands Back.
多处派活的代价,是每处都有活等你收。扫两眼 diff、看它说「测试过了」、合并 —— 那不是验收,是祈祷。这一章是全书的书眼:验收标准在派活时就写死,交付要带证据;本地 /review 用独立 reviewer 审 diff,应用里一字一行的小修直接行内编辑,要追根因的反馈留评论送回下一轮,侧栏 PR review 再把本地改动与 PR 收到一起;GitHub 上 @codex review 与 automatic reviews 负责自动底审。最后把反复出现的错误沉淀进 AGENTS.md 的 Review guidelines。The price of dispatching work across surfaces is work waiting to be collected from each one. Skimming a diff, reading its 'tests pass', merging — that's prayer, not review. This chapter is the book's centerpiece: acceptance criteria are fixed at dispatch and delivery carries evidence; local /review runs an independent reviewer; certain one-line fixes happen directly inside the app diff, comments return deeper work to the next turn, and side-panel PR review collects local changes and PRs together; on GitHub, @codex review and automatic reviews provide the baseline pass. Repeated mistakes then settle into AGENTS.md Review guidelines.
多处派活的代价,是每处都有活等你收:编辑器里一份 diff、云端一摞 PR、CI 里一条自动改动、定时任务的一份夜间班报。扫两眼、看它说「测试过了」、合并 —— 那不是验收,是祈祷。这一章是全书的书眼:会派活的人已经太多,能系统性验收 agent 的人还太少。它讲怎么把「看一眼就合」升级成一套真正的收活工作流,并接上《Agent 实战》里评测那一章 —— Codex 是你派活的地方,评测是你确认没被坑的地方。The price of dispatching across surfaces is work waiting to be collected from each one: a diff in the editor, a stack of PRs from the cloud, an automatic change in CI, a nightly shift report from a scheduled task. Skim it, read its 'tests pass', merge — that's prayer, not review. This chapter is the book's centerpiece: plenty of people can delegate to agents; few can systematically verify them. It upgrades 'glance and merge' into a real collection workflow, and connects to the evaluation chapter of Agent in Practice — Codex is where you delegate; evaluation is where you make sure you weren't burned.
— I
底线:一条能复跑的检查The Floor: One Rerunnable Check.
没有检查的时候,「看起来做完了」是它唯一的停机信号,而你是唯一的验证回路 —— 每个错误都躺在 diff 里,等你亲眼看见。官方最佳实践把这条说得很直:别停在让它改,让它写 / 更新测试、跑该跑的检查、确认结果、审自己 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠你的 prompt 或 AGENTS.md 告诉它。1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-08-05。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-08-05.有了能返回过 / 不过的信号,回路才闭合:它干活、跑检查、修到过为止。Without a check, 'looks done' is its only stopping signal, and you are the only verification loop — every mistake lies in the diff, waiting for your eyes. The official best practices put it plainly: don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review its own diff for bugs and regressions, then you accept — but it must know what 'good' looks like, told by your prompt or AGENTS.md.1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-08-05。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-08-05. Once there's a pass/fail signal it can read, the loop closes: it works, runs the check, iterates until green.纪律有两条。其一:验收标准在派活时就写死 —— 第 3 章那个四件套里的「完成标准」,就是为这一刻准备的。其二:要证据,不要断言。让它交测试输出、贴跑过的命令和返回,而不是一句「搞定了」;看证据比你亲手重跑一遍快,也是你敢不盯着它的前提。这两条尤其在云端与无人值守(第 8、9 章)时救命 —— 那些活你根本没在场看它跑。Two disciplines. First: acceptance criteria are fixed at dispatch — the 'done-when' in chapter 3's four parts exists for this moment. Second: evidence, not assertions. Have it hand over test output, the command it ran and what came back — not a 'done!'. Reading evidence is faster than rerunning the verification yourself, and it's what lets you stop watching. Both are lifesavers under cloud and unattended work (chapters 8, 9) — that work you never watched run.
提示词Prompt派活时把验收一起派下去Send the acceptance criteria with the job
这件活的验收标准:<一条可检验的标准>。
做完后跑 <检查命令>,把完整输出贴给我。
没验证过的部分明说「没验证」,不要写「应该可以」。
检查跑不过就修到跑过再交,别把失败的输出藏起来。
最后列出你改过的每个文件和一句为什么。Acceptance criteria for this task: <one checkable criterion>.
When done, run <check command> and paste the full output.
Mark anything unverified as 'not verified' — never write 'should work'.
If the check fails, fix until it passes before handing off; don't bury failing output.
End with every file you changed and one line on why.
这件活的验收标准:CSV 导出的用例全部通过,已有 JSON 导出测试仍然全绿。
做完后跑 pnpm test -- src/export,把完整输出贴给我。
没验证过的部分明说「没验证」,不要写「应该可以」。
检查跑不过就修到跑过再交,别把失败的输出藏起来。
最后列出你改过的每个文件和一句为什么。Acceptance criteria for this task: all CSV-export cases pass, and existing JSON-export tests stay green.
When done, run pnpm test -- src/export and paste the full output.
Mark anything unverified as 'not verified' — never write 'should work'.
If the check fails, fix until it passes before handing off; don't bury failing output.
End with every file you changed and one line on why.
— II
对着计划审,换个脑子审Against the Plan, with a Fresh Mind.
检查证明「它能跑」,证明不了「是你要的」。跑得过测试的代码,也可能顺手重构了你没让它动的模块、绕开了你点名的方案。第二层验收要有基线 —— 最现成的基线,是第 3 章计划模式里你批准过的那份计划:把 diff 对着它逐条核,要求都实现了吗、点名的边界情况有测试吗、范围之外的东西动了没有。1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-08-05。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-08-05.A check proves it runs; it doesn't prove it's what you asked for. Code that passes tests can still have refactored a module you never authorized or side-stepped the approach you named. The second layer needs a baseline — and the readiest one is the plan you approved in chapter 3's plan mode: read the diff against it item by item. Every requirement implemented? The named edge cases tested? Anything outside scope touched?1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-08-05。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-08-05.这层审最好换个脑子来。本地 /review 起一个专门 reviewer 读选定 diff、按优先级报告发现,而且不改工作区。2注 2Note 2OpenAI Codex 文档 · Code review —— 提交 / 推送前审代码。本地 /review 起一个专门的 reviewer 读选定的 diff、报按优先级排序的发现,不改你的工作区。应用里有 review pane:按 Unstaged / Staged / Commit / Branch / Last turn 看,可按文件 / 按 hunk stage 或 revert,行内评论会作上下文喂回下一轮 Codex。可设 review_model 让审查用不同模型,或设 detached 起独立审查对话。截至 2026-08-05。OpenAI Codex Docs · Code review — inspect code before you commit or push. Local /review starts a dedicated reviewer that reads the chosen diff and reports prioritized findings without changing your working tree. The app has a review pane: view by Unstaged / Staged / Commit / Branch / Last turn, stage or revert per file / per hunk, and inline comments feed as context into the next Codex turn. You can set review_model to review with a different model, or detached to start a separate review chat. As of 2026-08-05.第 6 章的自定义 reviewer 则让模型、推理力度、只读沙箱与 docs MCP 固定下来。应用内要分两类反馈:确定的一字一行小修,直接在 diff 里行内编辑;需要 agent 追原因、补测试的,留行内评论喂回下一轮。侧栏 PR review 把 PR 与本地改动放在同一收活位置,多仓库项目还能一次看齐各仓 diff。7注 7Note 7OpenAI · What's new / ChatGPT release notes —— 合并后的桌面 Codex 支持 diff 内联编辑和侧栏 PR review;多仓库项目随后支持在一次 review 中查看所有仓库的 diff。行内小修可直接改,需要 agent 推理的反馈再作为评论送回下一轮。截至 2026-08-05。OpenAI · What's new / ChatGPT release notes — merged desktop Codex supports inline diff editing and PR review in the side panel; multi-repository projects later added viewing all repository diffs in one review. Tiny line edits can be made directly, while feedback that needs agent reasoning returns as a comment for the next turn. As of 2026-08-05.This layer is best done by a different mind. Local /review starts a dedicated reviewer that reads the selected diff, prioritizes findings, and does not change the working tree.2注 2Note 2OpenAI Codex 文档 · Code review —— 提交 / 推送前审代码。本地 /review 起一个专门的 reviewer 读选定的 diff、报按优先级排序的发现,不改你的工作区。应用里有 review pane:按 Unstaged / Staged / Commit / Branch / Last turn 看,可按文件 / 按 hunk stage 或 revert,行内评论会作上下文喂回下一轮 Codex。可设 review_model 让审查用不同模型,或设 detached 起独立审查对话。截至 2026-08-05。OpenAI Codex Docs · Code review — inspect code before you commit or push. Local /review starts a dedicated reviewer that reads the chosen diff and reports prioritized findings without changing your working tree. The app has a review pane: view by Unstaged / Staged / Commit / Branch / Last turn, stage or revert per file / per hunk, and inline comments feed as context into the next Codex turn. You can set review_model to review with a different model, or detached to start a separate review chat. As of 2026-08-05. Chapter 6's custom reviewer fixes its model, reasoning, read-only sandbox, and docs MCP. In the app, split feedback in two: make a certain one-line correction directly inside the diff; leave an inline comment for anything that needs the agent to trace causes or add tests. Side-panel PR review keeps PR and local changes in one collection surface, and multi-repo projects can show every repository's diff in one review.7注 7Note 7OpenAI · What's new / ChatGPT release notes —— 合并后的桌面 Codex 支持 diff 内联编辑和侧栏 PR review;多仓库项目随后支持在一次 review 中查看所有仓库的 diff。行内小修可直接改,需要 agent 推理的反馈再作为评论送回下一轮。截至 2026-08-05。OpenAI · What's new / ChatGPT release notes — merged desktop Codex supports inline diff editing and PR review in the side panel; multi-repository projects later added viewing all repository diffs in one review. Tiny line edits can be made directly, while feedback that needs agent reasoning returns as a comment for the next turn. As of 2026-08-05.
— III
在 GitHub 上收一批,再把错固化Collect a Batch on GitHub, Then Codify the Miss.
第 8 章一批 PR 涌回来时,你需要的不是逐个手审,是一道自动底审。在 chatgpt.com/codex 的设置里为仓库开启后,PR 评论里 @codex review 就触发一次:Codex 先给个 👀、再像队友一样贴出审查,只标 P0 / P1。3注 3Note 3OpenAI Codex 文档 · Codex code review in GitHub —— 在 chatgpt.com/codex 的 code-review 设置里为仓库开启后,PR 评论里 @codex review 触发一次审查:Codex 先给 👀、再像队友一样贴一条审查,GitHub 上只标 P0 与 P1 高优先级问题。开 Automatic reviews 则每个新 PR 自动审、无需 @。审查跟随最近的 AGENTS.md 里的「Review guidelines」;可「@codex fix the P1 issue」让它起云端对话推修复。截至 2026-08-05。OpenAI Codex Docs · Codex code review in GitHub — after enabling it for a repo in code-review settings at chatgpt.com/codex, an @codex review comment on a PR triggers a review: Codex reacts 👀, then posts a review like a teammate, flagging only P0 and P1 high-priority issues on GitHub. Turning on Automatic reviews reviews every new PR without an @. The review follows the 'Review guidelines' in the nearest AGENTS.md; '@codex fix the P1 issue' has it start a cloud chat and push a fix. As of 2026-08-05.开 Automatic reviews,每个新 PR 都自动过一遍,不用你 @ —— 这正是 OpenAI 自己「审 100% 的 PR」的做法。1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-08-05。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-08-05.而且它跟着你最近的 AGENTS.md 里的 Review guidelines 走(第 5 章那段不是摆设);看到问题,一句 @codex fix the P1 issue 就让它起云端对话把修复推回分支。3注 3Note 3OpenAI Codex 文档 · Codex code review in GitHub —— 在 chatgpt.com/codex 的 code-review 设置里为仓库开启后,PR 评论里 @codex review 触发一次审查:Codex 先给 👀、再像队友一样贴一条审查,GitHub 上只标 P0 与 P1 高优先级问题。开 Automatic reviews 则每个新 PR 自动审、无需 @。审查跟随最近的 AGENTS.md 里的「Review guidelines」;可「@codex fix the P1 issue」让它起云端对话推修复。截至 2026-08-05。OpenAI Codex Docs · Codex code review in GitHub — after enabling it for a repo in code-review settings at chatgpt.com/codex, an @codex review comment on a PR triggers a review: Codex reacts 👀, then posts a review like a teammate, flagging only P0 and P1 high-priority issues on GitHub. Turning on Automatic reviews reviews every new PR without an @. The review follows the 'Review guidelines' in the nearest AGENTS.md; '@codex fix the P1 issue' has it start a cloud chat and push a fix. As of 2026-08-05.When a batch of PRs floods back from chapter 8, what you need isn't hand-reviewing each — it's an automatic baseline review. After enabling it for a repo in settings at chatgpt.com/codex, an @codex review comment on a PR triggers one: Codex reacts 👀, then posts a review like a teammate, flagging only P0 / P1.3注 3Note 3OpenAI Codex 文档 · Codex code review in GitHub —— 在 chatgpt.com/codex 的 code-review 设置里为仓库开启后,PR 评论里 @codex review 触发一次审查:Codex 先给 👀、再像队友一样贴一条审查,GitHub 上只标 P0 与 P1 高优先级问题。开 Automatic reviews 则每个新 PR 自动审、无需 @。审查跟随最近的 AGENTS.md 里的「Review guidelines」;可「@codex fix the P1 issue」让它起云端对话推修复。截至 2026-08-05。OpenAI Codex Docs · Codex code review in GitHub — after enabling it for a repo in code-review settings at chatgpt.com/codex, an @codex review comment on a PR triggers a review: Codex reacts 👀, then posts a review like a teammate, flagging only P0 and P1 high-priority issues on GitHub. Turning on Automatic reviews reviews every new PR without an @. The review follows the 'Review guidelines' in the nearest AGENTS.md; '@codex fix the P1 issue' has it start a cloud chat and push a fix. As of 2026-08-05. Turn on Automatic reviews and every new PR gets a pass without an @ — exactly how OpenAI itself 'reviews 100% of PRs.'1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-08-05。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-08-05. And it follows the Review guidelines in your nearest AGENTS.md (chapter 5's block wasn't decoration); on a problem, a single @codex fix the P1 issue has it start a cloud chat and push the fix back to the branch.3注 3Note 3OpenAI Codex 文档 · Codex code review in GitHub —— 在 chatgpt.com/codex 的 code-review 设置里为仓库开启后,PR 评论里 @codex review 触发一次审查:Codex 先给 👀、再像队友一样贴一条审查,GitHub 上只标 P0 与 P1 高优先级问题。开 Automatic reviews 则每个新 PR 自动审、无需 @。审查跟随最近的 AGENTS.md 里的「Review guidelines」;可「@codex fix the P1 issue」让它起云端对话推修复。截至 2026-08-05。OpenAI Codex Docs · Codex code review in GitHub — after enabling it for a repo in code-review settings at chatgpt.com/codex, an @codex review comment on a PR triggers a review: Codex reacts 👀, then posts a review like a teammate, flagging only P0 and P1 high-priority issues on GitHub. Turning on Automatic reviews reviews every new PR without an @. The review follows the 'Review guidelines' in the nearest AGENTS.md; '@codex fix the P1 issue' has it start a cloud chat and push a fix. As of 2026-08-05.验收做到第三遍,就该问:哪部分能固化?按约束力从软到硬,有三个台阶。最软是 AGENTS.md 的 Review guidelines:把「同一个错」写成一条审查规矩,下次自动审时就替你盯着 —— 但它是请求,不是保证(第 5 章那条记忆的边界)。硬一点是第 9 章的 hook:收工前必跑的校验,harness 执行、跳不过。最硬的一格,是把犯过的错写成一条测试:从此那个错不归你拦,归测试拦。By the third round of review, ask: which part can be codified? By grip, soft to hard, there are three steps. Softest is AGENTS.md's Review guidelines: turn 'the same mistake' into a review rule so the next automatic review watches for you — but it's a request, not a guarantee (chapter 5's memory boundary). Harder is chapter 9's hook: a check that must run before finishing, executed by the harness, unskippable. The hardest cell is turning a mistake it made into a test: from then on, that mistake is caught by the suite, not by you.
— IV
一次对了不算数:接上评测Once Right Isn't Enough: Enter Evaluation.
单件活的验收到此闭环。但多处自动化意味着你迟早在重复派同一类活 —— 每周的依赖升级、每个 PR 的安全审、每天的日志巡检。这时候该换单位了:单件问「这次对不对」,批量要问「这类活它多稳」。τ-bench 给这个直觉起了正式的名字:pass@k 量「k 次里至少对一次」,是能力上限;pass^k 量「k 次全对」,是可靠性。实测扎心:当时最强的 function calling agent 任务成功率不到 50%,retail 域连续 8 次全对的 pass^8 掉到 25% 以下。5注 5Note 5Yao et al. · 「τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains」(2024,arXiv:2406.12045)—— 提出 pass^k「度量 agent 行为在多次试验下的可靠性」(k 次全部成功的概率,区别于 pass@k 的至少一次);实测当时最强的 function calling agent(gpt-4o)任务成功率不到 50%,retail 域 pass^8 低于 25%。Yao et al. · 'τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains' (2024, arXiv:2406.12045) — proposes pass^k 'to evaluate the reliability of agent behavior over multiple trials' (the probability that all k trials succeed, vs pass@k's at-least-once); finds the strongest function-calling agents of the day (gpt-4o) succeed on under 50% of tasks, with pass^8 under 25% in retail.跑对一次和次次跑对之间,隔着「你敢不敢把它设成定时任务」的全部距离。For a single job, the loop closes here. But automation across surfaces means you're sooner or later delegating the same kind of job repeatedly — weekly dependency bumps, security review on every PR, daily log patrol. Time to change units: for one job you ask 'was this one right'; for a stream you ask 'how steady is it on this kind'. τ-bench gave that intuition its formal name: pass@k measures at-least-once-in-k — the capability ceiling; pass^k measures all-k-correct — reliability. The findings sting: the strongest function-calling agents of the day succeeded on under 50% of tasks, and in retail pass^8 — eight straight successes — fell below 25%.5注 5Note 5Yao et al. · 「τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains」(2024,arXiv:2406.12045)—— 提出 pass^k「度量 agent 行为在多次试验下的可靠性」(k 次全部成功的概率,区别于 pass@k 的至少一次);实测当时最强的 function calling agent(gpt-4o)任务成功率不到 50%,retail 域 pass^8 低于 25%。Yao et al. · 'τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains' (2024, arXiv:2406.12045) — proposes pass^k 'to evaluate the reliability of agent behavior over multiple trials' (the probability that all k trials succeed, vs pass@k's at-least-once); finds the strongest function-calling agents of the day (gpt-4o) succeed on under 50% of tasks, with pass^8 under 25% in retail. Between succeeding once and succeeding every time lies the whole distance of whether you dare put it on a schedule.第二个坑在「改进」上。你换了模型、改了 AGENTS.md、加了个 skill,感觉变好了 —— 感觉不算数。评测是实验,实验自带噪音:报成功率要带标准误;比较改动前后,用同一批任务的逐题配对差,而不是两个总分相减;想在 80% 功效下检出 3 个点的差距,大约要 1000 道题。6注 6Note 6Miller · 「Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations」(Anthropic,2024-11,arXiv:2411.00640)—— 把评测当实验:报成功率要带标准误;比较两个配置,用同一批题上的题级配对差做推断,而不是两个总分相减;动手前先算最小可检测效应(MDE)—— 论文算例:80% 功效下检出 3 个点的差距约需 1000 题。Miller · 'Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations' (Anthropic, 2024-11, arXiv:2411.00640) — treat evals as experiments: report success rates with standard errors; compare two configurations via inference on question-level paired differences over the same questions, not by subtracting two totals; compute the Minimum Detectable Effect first — the paper's worked example needs roughly 1,000 questions to detect a 3-point difference at 80% power.日常尺度记一条就够:小样本上的个位数点差,不构成「变好了」的结论 —— 它只构成「再跑几遍」的理由。The second trap is 'improvement'. You switched models, edited AGENTS.md, added a skill, and it feels better — feelings don't count. An eval is an experiment, and experiments carry noise: report success rates with standard errors; compare before and after via per-task paired differences on the same batch, not by subtracting two totals; detecting a 3-point difference at 80% power takes roughly a thousand questions.6注 6Note 6Miller · 「Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations」(Anthropic,2024-11,arXiv:2411.00640)—— 把评测当实验:报成功率要带标准误;比较两个配置,用同一批题上的题级配对差做推断,而不是两个总分相减;动手前先算最小可检测效应(MDE)—— 论文算例:80% 功效下检出 3 个点的差距约需 1000 题。Miller · 'Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations' (Anthropic, 2024-11, arXiv:2411.00640) — treat evals as experiments: report success rates with standard errors; compare two configurations via inference on question-level paired differences over the same questions, not by subtracting two totals; compute the Minimum Detectable Effect first — the paper's worked example needs roughly 1,000 questions to detect a 3-point difference at 80% power. At everyday scale, one line suffices: single-digit point differences on small samples are not a conclusion that it improved — only a reason to run it a few more times.落地不用自建 harness。把 Codex 坑过你的每个案例攒下来 —— 十几二十条,每条带上第 I 节那样的可检验标准,就是你的回归集;每次动配置,用 codex exec 把这批活重跑一遍、--output-schema 拿结构化结果对一对。4注 4Note 4OpenAI Codex 文档 · Non-interactive mode —— codex exec 非交互跑一次;--json 出 JSONL 事件流,--output-schema 让最终响应符合 JSON Schema,-o 写最终消息到文件;文档场景为脚本与 CI/CD。适合把一批攒下的回归用例重跑、拿结构化结果对比。截至 2026-08-05。OpenAI Codex Docs · Non-interactive mode — codex exec runs once non-interactively; --json emits a JSONL event stream, --output-schema makes the final response conform to a JSON Schema, and -o writes the final message to a file; the documented scenarios are scripts and CI/CD. Right for rerunning a saved batch of regression cases and diffing structured results. As of 2026-08-05.这正是《Agent 实战》里 golden set 加回归的家用版,也是两本书真正合流的地方:那本书教你把评测建成产线,这本书把它接到你每天派活的本地、云端与 GitHub —— Codex 是你派活的地方,评测是你确认没被坑的地方。Landing this needs no bespoke harness. Save every case where Codex burned you — fifteen or twenty, each with a checkable criterion like section I's — and that's your regression set; whenever you touch the configuration, rerun the batch with codex exec and diff the structured results via --output-schema.4注 4Note 4OpenAI Codex 文档 · Non-interactive mode —— codex exec 非交互跑一次;--json 出 JSONL 事件流,--output-schema 让最终响应符合 JSON Schema,-o 写最终消息到文件;文档场景为脚本与 CI/CD。适合把一批攒下的回归用例重跑、拿结构化结果对比。截至 2026-08-05。OpenAI Codex Docs · Non-interactive mode — codex exec runs once non-interactively; --json emits a JSONL event stream, --output-schema makes the final response conform to a JSON Schema, and -o writes the final message to a file; the documented scenarios are scripts and CI/CD. Right for rerunning a saved batch of regression cases and diffing structured results. As of 2026-08-05. This is the household edition of the golden-set-and-regression line in Agent in Practice — and where the two books truly meet: that one teaches you to build evaluation into a production line; this one plugs it into the local, cloud, and GitHub work you delegate every day. Codex is where you delegate; evaluation is where you make sure you weren't burned.动手 · 把这章装进你的下一次派活:Hands-on · build this chapter into your next delegation:
给你最近派出的一件活补一条能复跑的检查,并要它交证据(输出、命令)—— 不收「搞定了」。Retrofit one recently delegated job with a rerunnable check, and require evidence (output, command) — don't accept 'done'.
挑一份你本来打算直接合的 diff,先用 /review 对着计划审一遍,数数它找出几条你没看到的。Take a diff you were about to merge as-is, run /review against the plan first, and count what it finds that you missed.
把 Codex 最近一次坑你的错沉淀掉:进 AGENTS.md 的 Review guidelines、变成一条 hook,或写成一条测试。Settle Codex's most recent miss: into AGENTS.md's Review guidelines, into a hook, or into a test.
派活的人已经过剩,
验收的人还稀缺.
Delegators are already plentiful;
verifiers are still scarce.
Aklman Library
— 讨论Discussion
讨论Discussion.
评论区初始化中…Initializing comments…
01 / 01
没有匹配结果No matches.
换个关键词，或按 Esc 回到页面Try another keyword, or press Esc to return
