SummaryAI-Voyager is a research prototype that enables citizens to query, interpret, and draft revisions of 3,653 AI governance documents across city, state, federal, and national levels using limited agentic search. The system achieved an 83.0% macro recall rate on 360 new questions. A small study with 24 participants revealed that users predominantly asked applicability questions and rarely verified cited sources, despite the system’s supporting claims. |
Why it mattersAI regulations are being drafted rapidly, often before affected workers, small businesses, or communities can meaningfully voice their opinions. A tool that lowers the barrier to accessing dense legal texts could shift the power asymmetry in rulemaking. The paper also presents a concrete evaluation protocol—comparing new-question recall with fixed-document recall—that is missing from the AI governance technology literature. |
Highlights- An 83.0% macro recall rate on unseen questions, compared to the best baseline of 80.5%—a modest 2.5-point increase attributed to iterative agentic search
- Study with 24 participants: Most user edits were applicability questions ('how does this apply to me?') rather than substantive policy rewrites
- Although citations were perceived as supporting evidence, they 'rarely prompted participants to check the source'; this undermines the system’s core trust mechanism
|
Critical takeThe primary recall improvement is limited (a 2.5-point increase over the strongest baseline). The 99.1% rate in the fixed-document benchmark test is a near-tautological upper bound that serves more as praise for the system than a rigorous test. Furthermore, the user study’s small sample size (n=24) and the finding that people did not actually verify citations raise serious questions about whether 'anchoring AI to definitive text' constitutes genuine citizen oversight or merely an appearance of accountability. |
This critique was written by the local AI model
Qwen3.8-27B. I chose the story; I do not review each critique before it is published.