Release PR metadata rejected when it should be exempt, and @auto-review ignores user comments directing approval #287

New issue

Closed

opened 2026-06-13 05:22:29 -07:00 by jwilger · 0 comments

jwilger commented

2026-06-13 05:22:29 -07:00

Owner

Summary

Two recurring, related defects in @auto-review behavior, both reproduced again on Slipstream/eventcore#396 (see Slipstream/eventcore#396 (comment)). It should not have taken multiple tries to get an approval. Auto-review should have accepted the PR on the first pass; and even if it didn't, it should have approved after the first user comment saying the metadata is acceptable.

Problem A: Release PR title/description rejected when it should not be

Release PRs (e.g. title chore(release): v0.8.1, with a body that mirrors the title) are being failed under the PR metadata quality pre-merge check. On PR #396 the review reported:

PR metadata quality: failed

Rationale: The title does not follow the conventional commit format as it lacks a description after the version number. Additionally, the body is a copy of the title and does not provide a clear explanation of why the change is needed.

Offending text: chore(release): v0.8.1

This is wrong for release PRs. chore(release): vX.Y.Z is a valid, conventional title for an automated release, and a body that restates the release version is acceptable. The metadata check needs to recognize release PRs and exempt them from the "needs a descriptive summary / explain why" requirements, rather than treating them like normal feature/fix PRs.

This has been raised before; it is recurring and needs a durable fix (not another one-off).

Problem B: @auto-review does not respond naturally to user comments directed at it

When a user replies to @auto-review correcting it — e.g. on PR #396 the user wrote:

No, as I told you, this is acceptable for a release PR.

— auto-review should treat that as an authoritative human instruction and re-evaluate / approve accordingly. Instead it took multiple rounds to get an approval, and the agent does not appear to be incorporating the user's prior comments ("as I told you" implies it had already been told once). Directed comments should:

Be read and understood as instructions to the agent.
Override the agent's own metadata-quality objection when a human says the metadata is acceptable.
Result in approval on the next pass, without the user having to repeat themselves.

Acceptance criteria

A release PR with a title like chore(release): vX.Y.Z and a release-style body passes the PR metadata quality check on the first review pass (no human intervention required).
When a human comments on the PR telling @auto-review that the metadata/title is acceptable, the agent re-reviews and approves on the next pass, instead of repeating the same objection.
@auto-review's responses acknowledge and address the content of the user's comments directed at it.

References

Example: Slipstream/eventcore#396 (comment)
Auto-review's rejecting comment: Slipstream/eventcore#396 (comment)

## Summary Two recurring, related defects in @auto-review behavior, both reproduced again on https://git.johnwilger.com/Slipstream/eventcore/pulls/396 (see https://git.johnwilger.com/Slipstream/eventcore/pulls/396#issuecomment-9928). It should not have taken multiple tries to get an approval. Auto-review should have accepted the PR on the first pass; and even if it didn't, it should have approved after the first user comment saying the metadata is acceptable. ## Problem A: Release PR title/description rejected when it should not be Release PRs (e.g. title `chore(release): v0.8.1`, with a body that mirrors the title) are being failed under the **PR metadata quality** pre-merge check. On PR #396 the review reported: > * PR metadata quality: failed > * Rationale: The title does not follow the conventional commit format as it lacks a description after the version number. Additionally, the body is a copy of the title and does not provide a clear explanation of why the change is needed. > * Offending text: chore(release): v0.8.1 This is wrong for release PRs. `chore(release): vX.Y.Z` is a valid, conventional title for an automated release, and a body that restates the release version is acceptable. The metadata check needs to recognize release PRs and exempt them from the "needs a descriptive summary / explain why" requirements, rather than treating them like normal feature/fix PRs. This has been raised before; it is recurring and needs a durable fix (not another one-off). ## Problem B: @auto-review does not respond naturally to user comments directed at it When a user replies to @auto-review correcting it — e.g. on PR #396 the user wrote: > No, as I told you, this is acceptable for a release PR. — auto-review should treat that as an authoritative human instruction and re-evaluate / approve accordingly. Instead it took multiple rounds to get an approval, and the agent does not appear to be incorporating the user's prior comments ("as I told you" implies it had already been told once). Directed comments should: 1. Be read and understood as instructions to the agent. 2. Override the agent's own metadata-quality objection when a human says the metadata is acceptable. 3. Result in approval on the next pass, without the user having to repeat themselves. ## Acceptance criteria - A release PR with a title like `chore(release): vX.Y.Z` and a release-style body passes the PR metadata quality check on the first review pass (no human intervention required). - When a human comments on the PR telling @auto-review that the metadata/title is acceptable, the agent re-reviews and approves on the next pass, instead of repeating the same objection. - @auto-review's responses acknowledge and address the content of the user's comments directed at it. ## References - Example: https://git.johnwilger.com/Slipstream/eventcore/pulls/396#issuecomment-9928 - Auto-review's rejecting comment: https://git.johnwilger.com/Slipstream/eventcore/pulls/396#issuecomment-9927

jwilger added the

bug

label

2026-06-13 05:23:43 -07:00

jwilger referenced this issue from a commit

2026-06-13 08:01:51 -07:00

fix(review): exempt release PRs from metadata-quality check (#287)

jwilger referenced this issue from a commit

2026-06-13 08:01:51 -07:00

feat(config): add per-project override_approvers allow-list (#287)

jwilger referenced this issue from a commit

2026-06-13 08:01:51 -07:00