验收 · 怎么收 agent 交回的活Review · How to Collect What the Agent Hands Back.
四面开工的代价,是四面都有活等你收。扫两眼 diff、看它说「测试过了」、合并 —— 那不是验收,是祈祷。这一章是全书的书眼:验收标准在派活时就写死,交付要带证据;本地 /review 用一个独立的 reviewer 审 diff,应用里的 review pane 逐 hunk 收放、行内评论直接喂回下一轮;GitHub 上 @codex review 只报 P0/P1,automatic reviews 让每个 PR 都过一道 —— OpenAI 自己 100% 的 PR 都由 Codex 审。最后是沉淀:同一个错第二次出现,写进 AGENTS.md 的 Review guidelines,让下一次自动被拦住。它直接接上《Agent 实战》的评测那章。The price of four fronts is work waiting to be collected on all four. Skimming a diff, reading its 'tests pass', merging — that's prayer, not review. This chapter is the book's centerpiece: acceptance criteria are fixed at dispatch and delivery must carry evidence; local /review runs an independent reviewer over the diff; the app's review pane stages and reverts per hunk with inline comments fed straight into the next turn; on GitHub, @codex review flags only P0/P1 and automatic reviews put every PR through a pass — at OpenAI, Codex reviews 100% of PRs. Then the settling: the second time the same mistake appears, it goes into AGENTS.md's Review guidelines so the next one is caught automatically. This connects straight into the evaluation chapter of Agent in Practice.
10 分钟 · 初稿 2026.0710 Min · Drafted 2026.07
四面开工的代价,是四面都有活等你收:编辑器里一份 diff、云端一摞 PR、CI 里一条自动改动、定时任务的一份夜间班报。扫两眼、看它说「测试过了」、合并 —— 那不是验收,是祈祷。这一章是全书的书眼:会派活的人已经太多,能系统性验收 agent 的人还太少。它讲怎么把「看一眼就合」升级成一套真正的收活工作流,并接上《Agent 实战》里评测那一章 —— Codex 是你派活的地方,评测是你确认没被坑的地方。The price of four fronts is work waiting on all four: a diff in the editor, a stack of PRs from the cloud, an automatic change in CI, a nightly shift report from a scheduled task. Skim it, read its 'tests pass', merge — that's prayer, not review. This chapter is the book's centerpiece: plenty of people can delegate to agents; few can systematically verify them. It upgrades 'glance and merge' into a real collection workflow, and connects to the evaluation chapter of Agent in Practice — Codex is where you delegate; evaluation is where you make sure you weren't burned.
— I
底线:一条能复跑的检查The Floor: One Rerunnable Check.
没有检查的时候,「看起来做完了」是它唯一的停机信号,而你是唯一的验证回路 —— 每个错误都躺在 diff 里,等你亲眼看见。官方最佳实践把这条说得很直:别停在让它改,让它写 / 更新测试、跑该跑的检查、确认结果、审自己 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠你的 prompt 或 AGENTS.md 告诉它。1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-07-17。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-07-17.有了能返回过 / 不过的信号,回路才闭合:它干活、跑检查、修到过为止。Without a check, 'looks done' is its only stopping signal, and you are the only verification loop — every mistake lies in the diff, waiting for your eyes. The official best practices put it plainly: don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review its own diff for bugs and regressions, then you accept — but it must know what 'good' looks like, told by your prompt or AGENTS.md.1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-07-17。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-07-17. Once there's a pass/fail signal it can read, the loop closes: it works, runs the check, iterates until green.纪律有两条。其一:验收标准在派活时就写死 —— 第 3 章那个四件套里的「完成标准」,就是为这一刻准备的。其二:要证据,不要断言。让它交测试输出、贴跑过的命令和返回,而不是一句「搞定了」;看证据比你亲手重跑一遍快,也是你敢不盯着它的前提。这两条尤其在无人值守和云端(第 7、8 章)时救命 —— 那些活你根本没在场看它跑。Two disciplines. First: acceptance criteria are fixed at dispatch — the 'done-when' in chapter 3's four parts exists for this moment. Second: evidence, not assertions. Have it hand over test output, the command it ran and what came back — not a 'done!'. Reading evidence is faster than rerunning the verification yourself, and it's what lets you stop watching. Both are lifesavers under unattended and cloud work (chapters 7, 8) — that work you never watched run.
提示词Prompt派活时把验收一起派下去Send the acceptance criteria with the job
这件活的验收标准:<一条可检验的标准>。
做完后跑 <检查命令>,把完整输出贴给我。
没验证过的部分明说「没验证」,不要写「应该可以」。
检查跑不过就修到跑过再交,别把失败的输出藏起来。
最后列出你改过的每个文件和一句为什么。Acceptance criteria for this task: <one checkable criterion>.
When done, run <check command> and paste the full output.
Mark anything unverified as 'not verified' — never write 'should work'.
If the check fails, fix until it passes before handing off; don't bury failing output.
End with every file you changed and one line on why.
这件活的验收标准:CSV 导出的用例全部通过,已有 JSON 导出测试仍然全绿。
做完后跑 pnpm test -- src/export,把完整输出贴给我。
没验证过的部分明说「没验证」,不要写「应该可以」。
检查跑不过就修到跑过再交,别把失败的输出藏起来。
最后列出你改过的每个文件和一句为什么。Acceptance criteria for this task: all CSV-export cases pass, and existing JSON-export tests stay green.
When done, run pnpm test -- src/export and paste the full output.
Mark anything unverified as 'not verified' — never write 'should work'.
If the check fails, fix until it passes before handing off; don't bury failing output.
End with every file you changed and one line on why.
— II
对着计划审,换个脑子审Against the Plan, with a Fresh Mind.
检查证明「它能跑」,证明不了「是你要的」。跑得过测试的代码,也可能顺手重构了你没让它动的模块、绕开了你点名的方案。第二层验收要有基线 —— 最现成的基线,是第 3 章计划模式里你批准过的那份计划:把 diff 对着它逐条核,要求都实现了吗、点名的边界情况有测试吗、范围之外的东西动了没有。1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-07-17。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-07-17.A check proves it runs; it doesn't prove it's what you asked for. Code that passes tests can still have refactored a module you never authorized or side-stepped the approach you named. The second layer needs a baseline — and the readiest one is the plan you approved in chapter 3's plan mode: read the diff against it item by item. Every requirement implemented? The named edge cases tested? Anything outside scope touched?1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-07-17。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-07-17.这层审最好换个脑子来。Codex 内置了这套:本地 /review 起一个专门的 reviewer 读选定的 diff、报按优先级排序的发现,而且不改你的工作区 —— 它只挑毛病,不顺手动手。2注 2Note 2OpenAI Codex 文档 · Code review —— 提交 / 推送前审代码。本地 /review 起一个专门的 reviewer 读选定的 diff、报按优先级排序的发现,不改你的工作区。应用里有 review pane:按 Unstaged / Staged / Commit / Branch / Last turn 看,可按文件 / 按 hunk stage 或 revert,行内评论会作上下文喂回下一轮 Codex。可设 review_model 让审查用不同模型,或设 detached 起独立审查对话。截至 2026-07-17。OpenAI Codex Docs · Code review — inspect code before you commit or push. Local /review starts a dedicated reviewer that reads the chosen diff and reports prioritized findings without changing your working tree. The app has a review pane: view by Unstaged / Staged / Commit / Branch / Last turn, stage or revert per file / per hunk, and inline comments feed as context into the next Codex turn. You can set review_model to review with a different model, or detached to start a separate review chat. As of 2026-07-17.审的范围能选:对基线分支审(PR 式)、审未提交改动、审某个 commit,或按你给的自定义指令审。1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-07-17。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-07-17.应用里还有 review pane:按 hunk stage 或 revert,行内评论会作上下文喂回下一轮 —— 你圈一行说「这里漏了空值」,它下一轮就知道去哪改。2注 2Note 2OpenAI Codex 文档 · Code review —— 提交 / 推送前审代码。本地 /review 起一个专门的 reviewer 读选定的 diff、报按优先级排序的发现,不改你的工作区。应用里有 review pane:按 Unstaged / Staged / Commit / Branch / Last turn 看,可按文件 / 按 hunk stage 或 revert,行内评论会作上下文喂回下一轮 Codex。可设 review_model 让审查用不同模型,或设 detached 起独立审查对话。截至 2026-07-17。OpenAI Codex Docs · Code review — inspect code before you commit or push. Local /review starts a dedicated reviewer that reads the chosen diff and reports prioritized findings without changing your working tree. The app has a review pane: view by Unstaged / Staged / Commit / Branch / Last turn, stage or revert per file / per hunk, and inline comments feed as context into the next Codex turn. You can set review_model to review with a different model, or detached to start a separate review chat. As of 2026-07-17.This layer is best done by a different mind. Codex builds this in: local /review starts a dedicated reviewer that reads the chosen diff and reports prioritized findings, and it doesn't change your working tree — it only finds gaps, it doesn't act.2注 2Note 2OpenAI Codex 文档 · Code review —— 提交 / 推送前审代码。本地 /review 起一个专门的 reviewer 读选定的 diff、报按优先级排序的发现,不改你的工作区。应用里有 review pane:按 Unstaged / Staged / Commit / Branch / Last turn 看,可按文件 / 按 hunk stage 或 revert,行内评论会作上下文喂回下一轮 Codex。可设 review_model 让审查用不同模型,或设 detached 起独立审查对话。截至 2026-07-17。OpenAI Codex Docs · Code review — inspect code before you commit or push. Local /review starts a dedicated reviewer that reads the chosen diff and reports prioritized findings without changing your working tree. The app has a review pane: view by Unstaged / Staged / Commit / Branch / Last turn, stage or revert per file / per hunk, and inline comments feed as context into the next Codex turn. You can set review_model to review with a different model, or detached to start a separate review chat. As of 2026-07-17. The scope is selectable: against a base branch (PR-style), uncommitted changes, a commit, or your custom instructions.1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-07-17。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-07-17. The app also has a review pane: stage or revert per hunk, and inline comments feed as context into the next turn — circle a line saying 'null case missed here' and the next turn knows where to fix.2注 2Note 2OpenAI Codex 文档 · Code review —— 提交 / 推送前审代码。本地 /review 起一个专门的 reviewer 读选定的 diff、报按优先级排序的发现,不改你的工作区。应用里有 review pane:按 Unstaged / Staged / Commit / Branch / Last turn 看,可按文件 / 按 hunk stage 或 revert,行内评论会作上下文喂回下一轮 Codex。可设 review_model 让审查用不同模型,或设 detached 起独立审查对话。截至 2026-07-17。OpenAI Codex Docs · Code review — inspect code before you commit or push. Local /review starts a dedicated reviewer that reads the chosen diff and reports prioritized findings without changing your working tree. The app has a review pane: view by Unstaged / Staged / Commit / Branch / Last turn, stage or revert per file / per hunk, and inline comments feed as context into the next Codex turn. You can set review_model to review with a different model, or detached to start a separate review chat. As of 2026-07-17.
— III
在 GitHub 上收一批,再把错固化Collect a Batch on GitHub, Then Codify the Miss.
第 7 章一批 PR 涌回来时,你需要的不是逐个手审,是一道自动底审。在 chatgpt.com/codex 的设置里为仓库开启后,PR 评论里 @codex review 就触发一次:Codex 先给个 👀、再像队友一样贴出审查,只标 P0 / P1。3注 3Note 3OpenAI Codex 文档 · Codex code review in GitHub —— 在 chatgpt.com/codex 的 code-review 设置里为仓库开启后,PR 评论里 @codex review 触发一次审查:Codex 先给 👀、再像队友一样贴一条审查,GitHub 上只标 P0 与 P1 高优先级问题。开 Automatic reviews 则每个新 PR 自动审、无需 @。审查跟随最近的 AGENTS.md 里的「Review guidelines」;可「@codex fix the P1 issue」让它起云端对话推修复。截至 2026-07-17。OpenAI Codex Docs · Codex code review in GitHub — after enabling it for a repo in code-review settings at chatgpt.com/codex, an @codex review comment on a PR triggers a review: Codex reacts 👀, then posts a review like a teammate, flagging only P0 and P1 high-priority issues on GitHub. Turning on Automatic reviews reviews every new PR without an @. The review follows the 'Review guidelines' in the nearest AGENTS.md; '@codex fix the P1 issue' has it start a cloud chat and push a fix. As of 2026-07-17.开 Automatic reviews,每个新 PR 都自动过一遍,不用你 @ —— 这正是 OpenAI 自己「审 100% 的 PR」的做法。1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-07-17。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-07-17.而且它跟着你最近的 AGENTS.md 里的 Review guidelines 走(第 5 章那段不是摆设);看到问题,一句 @codex fix the P1 issue 就让它起云端对话把修复推回分支。3注 3Note 3OpenAI Codex 文档 · Codex code review in GitHub —— 在 chatgpt.com/codex 的 code-review 设置里为仓库开启后,PR 评论里 @codex review 触发一次审查:Codex 先给 👀、再像队友一样贴一条审查,GitHub 上只标 P0 与 P1 高优先级问题。开 Automatic reviews 则每个新 PR 自动审、无需 @。审查跟随最近的 AGENTS.md 里的「Review guidelines」;可「@codex fix the P1 issue」让它起云端对话推修复。截至 2026-07-17。OpenAI Codex Docs · Codex code review in GitHub — after enabling it for a repo in code-review settings at chatgpt.com/codex, an @codex review comment on a PR triggers a review: Codex reacts 👀, then posts a review like a teammate, flagging only P0 and P1 high-priority issues on GitHub. Turning on Automatic reviews reviews every new PR without an @. The review follows the 'Review guidelines' in the nearest AGENTS.md; '@codex fix the P1 issue' has it start a cloud chat and push a fix. As of 2026-07-17.When a batch of PRs floods back from chapter 7, what you need isn't hand-reviewing each — it's an automatic baseline review. After enabling it for a repo in settings at chatgpt.com/codex, an @codex review comment on a PR triggers one: Codex reacts 👀, then posts a review like a teammate, flagging only P0 / P1.3注 3Note 3OpenAI Codex 文档 · Codex code review in GitHub —— 在 chatgpt.com/codex 的 code-review 设置里为仓库开启后,PR 评论里 @codex review 触发一次审查:Codex 先给 👀、再像队友一样贴一条审查,GitHub 上只标 P0 与 P1 高优先级问题。开 Automatic reviews 则每个新 PR 自动审、无需 @。审查跟随最近的 AGENTS.md 里的「Review guidelines」;可「@codex fix the P1 issue」让它起云端对话推修复。截至 2026-07-17。OpenAI Codex Docs · Codex code review in GitHub — after enabling it for a repo in code-review settings at chatgpt.com/codex, an @codex review comment on a PR triggers a review: Codex reacts 👀, then posts a review like a teammate, flagging only P0 and P1 high-priority issues on GitHub. Turning on Automatic reviews reviews every new PR without an @. The review follows the 'Review guidelines' in the nearest AGENTS.md; '@codex fix the P1 issue' has it start a cloud chat and push a fix. As of 2026-07-17. Turn on Automatic reviews and every new PR gets a pass without an @ — exactly how OpenAI itself 'reviews 100% of PRs.'1注 1Note 1OpenAI Codex 文档 · Best practices —— 别停在让它改;让它写 / 更新测试、跑该跑的检查、确认结果、审 diff 里的 bug 与回归,再由你接受 —— 但它得知道「好」长什么样,这靠 prompt 或 AGENTS.md。/review 给几种审法:对基线分支审(PR 式)、审未提交改动、审某个 commit、按自定义指令审。原话:「在 OpenAI,Codex 审 100% 的 PR。」截至 2026-07-17。OpenAI Codex Docs · Best practices — don't stop at asking it to change; have it write / update tests, run the right checks, confirm results, review the diff for bugs and regressions, then you accept — but it must know what 'good' looks like, from the prompt or AGENTS.md. /review offers several scopes: against a base branch (PR-style), uncommitted changes, a commit, or custom instructions. Verbatim: 'At OpenAI, Codex reviews 100% of PRs.' As of 2026-07-17. And it follows the Review guidelines in your nearest AGENTS.md (chapter 5's block wasn't decoration); on a problem, a single @codex fix the P1 issue has it start a cloud chat and push the fix back to the branch.3注 3Note 3OpenAI Codex 文档 · Codex code review in GitHub —— 在 chatgpt.com/codex 的 code-review 设置里为仓库开启后,PR 评论里 @codex review 触发一次审查:Codex 先给 👀、再像队友一样贴一条审查,GitHub 上只标 P0 与 P1 高优先级问题。开 Automatic reviews 则每个新 PR 自动审、无需 @。审查跟随最近的 AGENTS.md 里的「Review guidelines」;可「@codex fix the P1 issue」让它起云端对话推修复。截至 2026-07-17。OpenAI Codex Docs · Codex code review in GitHub — after enabling it for a repo in code-review settings at chatgpt.com/codex, an @codex review comment on a PR triggers a review: Codex reacts 👀, then posts a review like a teammate, flagging only P0 and P1 high-priority issues on GitHub. Turning on Automatic reviews reviews every new PR without an @. The review follows the 'Review guidelines' in the nearest AGENTS.md; '@codex fix the P1 issue' has it start a cloud chat and push a fix. As of 2026-07-17.验收做到第三遍,就该问:哪部分能固化?按约束力从软到硬,有三个台阶。最软是 AGENTS.md 的 Review guidelines:把「同一个错」写成一条审查规矩,下次自动审时就替你盯着 —— 但它是请求,不是保证(第 5 章那条记忆的边界)。硬一点是第 8 章的 hook:收工前必跑的校验,harness 执行、跳不过。最硬的一格,是把犯过的错写成一条测试:从此那个错不归你拦,归测试拦。By the third round of review, ask: which part can be codified? By grip, soft to hard, there are three steps. Softest is AGENTS.md's Review guidelines: turn 'the same mistake' into a review rule so the next automatic review watches for you — but it's a request, not a guarantee (chapter 5's memory boundary). Harder is chapter 8's hook: a check that must run before finishing, executed by the harness, unskippable. The hardest cell is turning a mistake it made into a test: from then on, that mistake is caught by the suite, not by you.
— IV
一次对了不算数:接上评测Once Right Isn't Enough: Enter Evaluation.
单件活的验收到此闭环。但四面开工意味着你迟早在重复派同一类活 —— 每周的依赖升级、每个 PR 的安全审、每天的日志巡检。这时候该换单位了:单件问「这次对不对」,批量要问「这类活它多稳」。τ-bench 给这个直觉起了正式的名字:pass@k 量「k 次里至少对一次」,是能力上限;pass^k 量「k 次全对」,是可靠性。实测扎心:当时最强的 function calling agent 任务成功率不到 50%,retail 域连续 8 次全对的 pass^8 掉到 25% 以下。5注 5Note 5Yao et al. · 「τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains」(2024,arXiv:2406.12045)—— 提出 pass^k「度量 agent 行为在多次试验下的可靠性」(k 次全部成功的概率,区别于 pass@k 的至少一次);实测当时最强的 function calling agent(gpt-4o)任务成功率不到 50%,retail 域 pass^8 低于 25%。Yao et al. · 'τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains' (2024, arXiv:2406.12045) — proposes pass^k 'to evaluate the reliability of agent behavior over multiple trials' (the probability that all k trials succeed, vs pass@k's at-least-once); finds the strongest function-calling agents of the day (gpt-4o) succeed on under 50% of tasks, with pass^8 under 25% in retail.跑对一次和次次跑对之间,隔着「你敢不敢把它设成定时任务」的全部距离。For a single job, the loop closes here. But four fronts means you're sooner or later delegating the same kind of job repeatedly — weekly dependency bumps, security review on every PR, daily log patrol. Time to change units: for one job you ask 'was this one right'; for a stream you ask 'how steady is it on this kind'. τ-bench gave that intuition its formal name: pass@k measures at-least-once-in-k — the capability ceiling; pass^k measures all-k-correct — reliability. The findings sting: the strongest function-calling agents of the day succeeded on under 50% of tasks, and in retail pass^8 — eight straight successes — fell below 25%.5注 5Note 5Yao et al. · 「τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains」(2024,arXiv:2406.12045)—— 提出 pass^k「度量 agent 行为在多次试验下的可靠性」(k 次全部成功的概率,区别于 pass@k 的至少一次);实测当时最强的 function calling agent(gpt-4o)任务成功率不到 50%,retail 域 pass^8 低于 25%。Yao et al. · 'τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains' (2024, arXiv:2406.12045) — proposes pass^k 'to evaluate the reliability of agent behavior over multiple trials' (the probability that all k trials succeed, vs pass@k's at-least-once); finds the strongest function-calling agents of the day (gpt-4o) succeed on under 50% of tasks, with pass^8 under 25% in retail. Between succeeding once and succeeding every time lies the whole distance of whether you dare put it on a schedule.第二个坑在「改进」上。你换了模型、改了 AGENTS.md、加了个 skill,感觉变好了 —— 感觉不算数。评测是实验,实验自带噪音:报成功率要带标准误;比较改动前后,用同一批任务的逐题配对差,而不是两个总分相减;想在 80% 功效下检出 3 个点的差距,大约要 1000 道题。6注 6Note 6Miller · 「Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations」(Anthropic,2024-11,arXiv:2411.00640)—— 把评测当实验:报成功率要带标准误;比较两个配置,用同一批题上的题级配对差做推断,而不是两个总分相减;动手前先算最小可检测效应(MDE)—— 论文算例:80% 功效下检出 3 个点的差距约需 1000 题。Miller · 'Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations' (Anthropic, 2024-11, arXiv:2411.00640) — treat evals as experiments: report success rates with standard errors; compare two configurations via inference on question-level paired differences over the same questions, not by subtracting two totals; compute the Minimum Detectable Effect first — the paper's worked example needs roughly 1,000 questions to detect a 3-point difference at 80% power.日常尺度记一条就够:小样本上的个位数点差,不构成「变好了」的结论 —— 它只构成「再跑几遍」的理由。The second trap is 'improvement'. You switched models, edited AGENTS.md, added a skill, and it feels better — feelings don't count. An eval is an experiment, and experiments carry noise: report success rates with standard errors; compare before and after via per-task paired differences on the same batch, not by subtracting two totals; detecting a 3-point difference at 80% power takes roughly a thousand questions.6注 6Note 6Miller · 「Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations」(Anthropic,2024-11,arXiv:2411.00640)—— 把评测当实验:报成功率要带标准误;比较两个配置,用同一批题上的题级配对差做推断,而不是两个总分相减;动手前先算最小可检测效应(MDE)—— 论文算例:80% 功效下检出 3 个点的差距约需 1000 题。Miller · 'Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations' (Anthropic, 2024-11, arXiv:2411.00640) — treat evals as experiments: report success rates with standard errors; compare two configurations via inference on question-level paired differences over the same questions, not by subtracting two totals; compute the Minimum Detectable Effect first — the paper's worked example needs roughly 1,000 questions to detect a 3-point difference at 80% power. At everyday scale, one line suffices: single-digit point differences on small samples are not a conclusion that it improved — only a reason to run it a few more times.落地不用自建 harness。把 Codex 坑过你的每个案例攒下来 —— 十几二十条,每条带上第 I 节那样的可检验标准,就是你的回归集;每次动配置,用 codex exec 把这批活重跑一遍、--output-schema 拿结构化结果对一对。4注 4Note 4OpenAI Codex 文档 · Non-interactive mode —— codex exec 非交互跑一次;--json 出 JSONL 事件流,--output-schema 让最终响应符合 JSON Schema,-o 写最终消息到文件;文档场景为脚本与 CI/CD。适合把一批攒下的回归用例重跑、拿结构化结果对比。截至 2026-07-17。OpenAI Codex Docs · Non-interactive mode — codex exec runs once non-interactively; --json emits a JSONL event stream, --output-schema makes the final response conform to a JSON Schema, and -o writes the final message to a file; the documented scenarios are scripts and CI/CD. Right for rerunning a saved batch of regression cases and diffing structured results. As of 2026-07-17.这正是《Agent 实战》里 golden set 加回归的家用版,也是两本书真正合流的地方:那本书教你把评测建成产线,这本书把它接到你每天派活的四个面上 —— Codex 是你派活的地方,评测是你确认没被坑的地方。Landing this needs no bespoke harness. Save every case where Codex burned you — fifteen or twenty, each with a checkable criterion like section I's — and that's your regression set; whenever you touch the configuration, rerun the batch with codex exec and diff the structured results via --output-schema.4注 4Note 4OpenAI Codex 文档 · Non-interactive mode —— codex exec 非交互跑一次;--json 出 JSONL 事件流,--output-schema 让最终响应符合 JSON Schema,-o 写最终消息到文件;文档场景为脚本与 CI/CD。适合把一批攒下的回归用例重跑、拿结构化结果对比。截至 2026-07-17。OpenAI Codex Docs · Non-interactive mode — codex exec runs once non-interactively; --json emits a JSONL event stream, --output-schema makes the final response conform to a JSON Schema, and -o writes the final message to a file; the documented scenarios are scripts and CI/CD. Right for rerunning a saved batch of regression cases and diffing structured results. As of 2026-07-17. This is the household edition of the golden-set-and-regression line in Agent in Practice — and where the two books truly meet: that one teaches you to build evaluation into a production line; this one plugs it into the four fronts you delegate from every day. Codex is where you delegate; evaluation is where you make sure you weren't burned.动手 · 把这章装进你的下一次派活:Hands-on · build this chapter into your next delegation:
给你最近派出的一件活补一条能复跑的检查,并要它交证据(输出、命令)—— 不收「搞定了」。Retrofit one recently delegated job with a rerunnable check, and require evidence (output, command) — don't accept 'done'.
挑一份你本来打算直接合的 diff,先用 /review 对着计划审一遍,数数它找出几条你没看到的。Take a diff you were about to merge as-is, run /review against the plan first, and count what it finds that you missed.
把 Codex 最近一次坑你的错沉淀掉:进 AGENTS.md 的 Review guidelines、变成一条 hook,或写成一条测试。Settle Codex's most recent miss: into AGENTS.md's Review guidelines, into a hook, or into a test.
派活的人已经过剩,
验收的人还稀缺.
Delegators are already plentiful;
verifiers are still scarce.
Aklman Library
— 讨论Discussion
讨论Discussion.
评论区初始化中…Initializing comments…
01 / 01
没有匹配结果No matches.
换个关键词，或按 Esc 回到页面Try another keyword, or press Esc to return