arXiv· Lorenzo Masoero·· 1 天前精选AI 评分54
TriAgent 在 43 天影子运行中处理 499 张工单,找出 92 处文档缺口
Mind the Gap: A Deployed Multi-Agent System for Documentation Maintenance
AI 导读
研究团队部署了以分诊智能体 TriAgent 为核心的多智能体系统,在 43 天影子模式运行中处理 499 张客服工单,找出 92 处文档缺口。由独立模型家族的 LLM 评委组成的评审组判定,TriAgent 可核查断言中 81.1% 正确(302 条中 245 条);在长度可比的回答对上,评审组 70.5% 偏好 TriAgent 的回答,23.2% 偏好人工工程师的回答。
变化判断
一个多智能体系统在 43 天影子运行中处理 499 张工单、找出 92 处文档缺口,人工复核后修补,修补后同一工单的回答在盲评中 71.4% 优于修补前。
正文 · 原文
Abstract:The "AI revolution" has enabled teams across the technology industry to deliver software features at unprecedented velocity. While code evolves faster than ever, customer-facing documentation often struggles to stay up to speed. As users increasingly rely on LLM-based agents that consume documentation as their primary interface, the impact of this documentation gap worsens: stale information propagates and is treated as authoritative. Signals for identifying where documentation has fallen behind already exist in abundance, both explicit (e.g., resolved support tickets revealing what documentation lacks) and implicit (e.g., code-documentation discrepancies detectable from automated loops). We deploy a multi-agent system, built around a triage agent we call "TriAgent," that continuously extracts these signals and repairs documentation drift. Over 43 days of shadow-mode operation, our system triaged 499 tickets and surfaced 92 gaps in the documentation. Using a panel of LLM judges from independent model families, 81.1% of TriAgent's verifiable assertions were correct (245 of the 302 assertions the human resolution addresses; majority of three judges); on length-comparable pairs the panel preferred TriAgent's response in 70.5% of pairs and the human engineer's in 23.2% (the rest had no judge majority). Importantly, agent- and human-sourced replies tended to be complementary, with the human often surfacing insights that were either buried within the existing documentation or missing from it entirely; our system turns each such insight into a documentation gap and, after human review, a fix. Repairing a gap measurably improved the agent's later answer: when the agent re-answered the same ticket after the repair, blinded judges preferred the post-repair answer over its own pre-repair answer 71.4% of the time (n=35). A blinded human audit confirms strong alignment on self-improvement.
| Comments: | Accepted at IAAI-27 (Innovative Applications of AI, co-located with AAAI 2027), Emerging Applications track. 12 pages (6 pages + references and appendix), 2 figures, 6 tables |
| Subjects: | Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL) |
| Cite as: | arXiv:2611.23142 [cs.SE] |
| (or arXiv:2611.23142v1 [cs.SE] for this version) | |
| https://doi.org/10.48550/arXiv.2611.23142 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Lorenzo Masoero [view email]
[v1]
Tue, 17 Nov 2026 17:59:13 UTC (45 KB)
来源:arXiv · arxiv.org