AI and human code review: comment counts do not establish quality
Separate automated feedback from maintainer acceptance using the same diff, and evaluate real defects, false positives and suggested fixes alongside review effort.
Choose an approach
AI review can help find issues and explain changes, while humans judge business intent and accepted risk. Have both inspect the same diff and require locatable evidence for each finding. More comments do not prove more real bugs, and silence is not approval. Decide which checks are reusable and who confirms that the final change is correct.
Automated review feedback can contain false positives
GitHub positions Copilot review as a supplement to humans and warns of missed defects and false positives. Suggested code needs validation even when attached to exact lines. Humans can incorporate product constraints absent from code but also overlook details, so automated findings can supply candidates. Separate style preferences from behavioral defects and require triggering inputs and consequences rather than counting debatable style as serious bugs.
Reproduce findings against the same diff
Use one fixed diff with identical requirements and context. Record findings independently, then have a maintainer verify each and retain minimal inputs or failing tests for reproducible defects. Test suggested repairs against other callers, not just the flagged line. Unreproduced concerns can remain open without being counted as confirmed discoveries. Record false-positive review time and whether irrelevant comments hide real issues to assess benefit for the team.
Assign validation responsibility and long-term costs
Repository review guidance should state business invariants and required validation, not demand more comments. Replay known defective and clean examples after model or prompt changes to monitor misses and false alarms. Include maintainer context and verification effort alongside fees. Apply the project’s existing tests and review rules before release and preserve approval evidence. This article checks official limitations without measuring detection rates or replacing human responsibility with automated feedback.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.