jwilger/auto_review
1
0
Fork 0
Code Issues 6 Pull requests 1 Releases Packages 1 Activity Actions

Qualitative pre-merge check on PR title and description #15

New issue
Open
opened 2026-05-01 14:19:15 -07:00 by auto-review · 0 comments
auto-review commented 2026-05-01 14:19:15 -07:00
Collaborator
Copy link

Today the bot has pre-merge checks for CHANGELOG, tests, and no-new-todos, but nothing about the PR-author-provided title and description. A PR with title "fix" and an empty description gets the same treatment as a well-crafted one.

Proposed: add a qualitative pre-merge check (cheap-tier LLM, like the natural-language custom checks already do). Pass criteria, off the top of head:

  • Title: imperative-mood verb at the start, scope marker (fix(area):, feat(area):, etc.) where the project's commit conventions use them, ≤72 chars.
  • Description: not empty, not just a copy of the title, addresses the why (links to issue/ticket/incident, or one sentence motivating the change).

When the check fails, surface it the same way the other pre-merge checks do — render in the review body's "Pre-merge checks" section. Combined with #7 (pre-merge fail → RequestChanges), a low-quality PR description would block merging until the author improves it.

This is advisory and configurable: opinionated style choices live in the repo's .auto_review.yaml (pre_merge_checks: already supports natural-language entries — this could just be the canonical default set, or a flag to enable it).

Calibration risk: cheap-tier LLMs over-flag. Anchor the prompt to specific failure modes ("title is just a verb without context") and require the model to quote the offending portion verbatim, so authors see exactly what tripped the check.

Today the bot has pre-merge checks for CHANGELOG, tests, and no-new-todos, but nothing about the PR-author-provided title and description. A PR with title "fix" and an empty description gets the same treatment as a well-crafted one. **Proposed:** add a qualitative pre-merge check (cheap-tier LLM, like the natural-language custom checks already do). Pass criteria, off the top of head: - **Title**: imperative-mood verb at the start, scope marker (`fix(area):`, `feat(area):`, etc.) where the project's commit conventions use them, ≤72 chars. - **Description**: not empty, not just a copy of the title, addresses the *why* (links to issue/ticket/incident, or one sentence motivating the change). When the check fails, surface it the same way the other pre-merge checks do — render in the review body's "Pre-merge checks" section. Combined with #7 (pre-merge fail → RequestChanges), a low-quality PR description would block merging until the author improves it. This is *advisory* and *configurable*: opinionated style choices live in the repo's `.auto_review.yaml` (`pre_merge_checks:` already supports natural-language entries — this could just be the canonical default set, or a flag to enable it). Calibration risk: cheap-tier LLMs over-flag. Anchor the prompt to specific failure modes ("title is just a verb without context") and require the model to quote the offending portion verbatim, so authors see exactly what tripped the check.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
issue-10-mandatory-sandbox
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
jwilger/auto_review#15
No description provided.