Bug: Auto Review replies to direct feedback without acknowledging or addressing it #274

New issue

Closed

opened 2026-05-19 14:59:02 -07:00 by jwilger · 0 comments

jwilger commented

2026-05-19 14:59:02 -07:00

Owner

Problem

Auto Review can respond to direct user feedback/comments without acknowledging or addressing the user's actual concern.

Observed on Slipstream/eddy PR #4:

User comment: Slipstream/eddy#4 (comment)
Follow-up user comment: Slipstream/eddy#4 (comment)
Bot replies/comments around that thread: Slipstream/eddy#4 (comment) and Slipstream/eddy#4 (comment)

The user pointed out that Auto Review's metadata failure was self-contradictory:

It literally meets the "should start with a verb like 'Add'" criteria.

Then followed up:

OK, but since the pr metadata isn't actually a problem, I'd sure appreciate an approval.

Instead of acknowledging the correction, explaining whether it could revise the prior result, or approving/clearing the metadata concern, Auto Review responded with generic summaries of the diff.

Expected behavior

When directly mentioned in a PR comment, especially when the user challenges or corrects a prior Auto Review finding, Auto Review should:

Explicitly acknowledge the user's comment.
Address the specific concern raised.
If the user is right, say so and revise/clear the finding when possible.
If the bot cannot change a prior review/status, explain that limitation and what the user can do next.
Avoid replying with an unrelated generic diff summary.

Actual behavior

Auto Review posted responses that looked superficially responsive but did not address or acknowledge the user's comments.

Impact

This is frustrating and erodes trust: users feel ignored even though the bot is actively replying. It also leaves incorrect review outcomes unresolved.

## Problem Auto Review can respond to direct user feedback/comments without acknowledging or addressing the user's actual concern. Observed on Slipstream/eddy PR #4: - User comment: https://git.johnwilger.com/Slipstream/eddy/pulls/4#issuecomment-8754 - Follow-up user comment: https://git.johnwilger.com/Slipstream/eddy/pulls/4#issuecomment-8756 - Bot replies/comments around that thread: https://git.johnwilger.com/Slipstream/eddy/pulls/4#issuecomment-8755 and https://git.johnwilger.com/Slipstream/eddy/pulls/4#issuecomment-8757 The user pointed out that Auto Review's metadata failure was self-contradictory: > It *literally* meets the "should start with a verb like 'Add'" criteria. Then followed up: > OK, but since the pr metadata isn't actually a problem, I'd sure appreciate an approval. Instead of acknowledging the correction, explaining whether it could revise the prior result, or approving/clearing the metadata concern, Auto Review responded with generic summaries of the diff. ## Expected behavior When directly mentioned in a PR comment, especially when the user challenges or corrects a prior Auto Review finding, Auto Review should: 1. Explicitly acknowledge the user's comment. 2. Address the specific concern raised. 3. If the user is right, say so and revise/clear the finding when possible. 4. If the bot cannot change a prior review/status, explain that limitation and what the user can do next. 5. Avoid replying with an unrelated generic diff summary. ## Actual behavior Auto Review posted responses that looked superficially responsive but did not address or acknowledge the user's comments. ## Impact This is frustrating and erodes trust: users feel ignored even though the bot is actively replying. It also leaves incorrect review outcomes unresolved.

jwilger added this to the 1.0 milestone

2026-05-19 16:32:46 -07:00

jwilger added the

bug

label

2026-05-19 16:33:04 -07:00

jwilger referenced this issue from a pull request that will close it,

2026-05-19 21:14:12 -07:00

fix(review): handle imperative metadata and quoted feedback #279

jwilger closed this issue

2026-05-19 21:16:28 -07:00

jwilger referenced this issue from a commit