The Reviewer That Can’t Read the Ticket

A few weeks ago I picked up a ticket that looked like twenty minutes of work. Four acceptance criteria, one file touched, nothing that needed a design conversation. I read the criteria fast, built the thing, opened the PR, and had an automated review back within a minute: clean implementation, consistent with existing patterns, tests look good. I was one click from merging it.

It was wrong. Not broken, not buggy, not badly written. Wrong in the way that costs the most to fix, which is that it did something adjacent to what was asked for. The third acceptance criterion said one thing and I had quietly built the other. I only found out because a colleague reviewed it, opened the ticket, and asked whether criterion three said what I thought it said. It did not.

The bot did not miss that. It never had a chance at it. The criteria lived in Jira. The review lived in GitHub. Nothing connected the two, so the review was answering a different question than the one I needed answered.

A human did catch it. That is the part I keep coming back to, because nothing in the pipeline earned it.

The review had everything except the point

There are three kinds of context a code review can draw on.

The diff: what changed, line by line. The repository: how we write code here, which conventions we follow, which patterns are idiomatic. And the intent: what was actually asked for, and how we will know when it is done.

Automated review is genuinely strong at the first two. It reads a diff more patiently than I do, and it knows the codebase’s conventions better than most of the team. What it cannot do is see the third, because the third is not in the repository. It is in a ticket, in a system the reviewer has never been introduced to.

So it answers the question it can answer. Is this well made? And the whole team hears a different question. Is this right?

Key Takeaway: An automated reviewer compares your code against the codebase. Only a connected one can compare it against the requirement.

The expensive part is the confidence

If the bot had simply said nothing useful, this would be a mild disappointment. The problem is that it says something confident.

Well structured. Idiomatic. Good test coverage. That is a verdict on craftsmanship, and it arrives in the same tone we use to endorse correctness. Nobody reads “follows existing patterns” and thinks “but is it the right pattern?” The green check spends the team’s skepticism, and there is a finite amount of it.

In my case the tests were green too, which made it worse. I had written them against my own misreading, so the suite did not catch the error. It encoded it. Green CI, green automated review, clean diff, small change. Four signals saying the same thing: nothing here needs a second look.

What stopped it was a person. Not a process. A colleague on a required human review, who chose to open the ticket before approving a twenty line change, at a moment when three green checks were telling them it was fine. We had that requirement on that repo, and on this occasion it held. But it held because of what one person decided to do with their afternoon, not because anything in the pipeline made it happen. If they had been busy, it would have shipped.

And notice what the automated review did for that reviewer. Nothing. It did not shorten their work or focus it. It added a confident thumbs up they had to discount before they could start the part that actually mattered.

That is the failure mode worth naming, because it is not “the AI missed a bug.” It is that the AI made being wrong feel fine, right up until a person overruled it.

Small tickets are where this lives

The instinct is to worry about big PRs. They are hard to review, so that is where we assume the risk is. In practice they are the safest kind, because they attract scrutiny. People block out time. They ask questions. They open the ticket.

A twenty line PR gets a scroll and an approve. It is small, it looks clean, the bot approved it, and there is a queue of other things. But on a small change the ticket is the specification. There is not enough code for the intent to be self-evident from reading it, so the entire meaning of the diff lives in a document the reviewer is not looking at.

Risk concentrates exactly where attention is thinnest. And it is your fastest, most trusted people generating most of those changes, all day, at volume.

Key Takeaway: The smaller the change, the more the requirement matters and the less anyone checks it.

This is plumbing, not model quality

Here is the part I want to be precise about, because the easy conclusion is wrong.

A better model does not fix this. A model that has not been shown the acceptance criteria cannot infer them, no matter how capable it is. It can guess from the branch name and the PR title, and guessing is what makes it dangerous, because a plausible guess arrives wrapped in the same confident tone as a verified fact.

The fix exists. Jira talks to GitHub. You can pull the ticket into the PR body automatically on open. You can connect the ticketing system to the agent that runs the review, so it reads the criteria before it reads the diff. You can tell the reviewer, in the repository’s own instructions, to check each criterion explicitly, and to say “I cannot verify intent” when nothing is linked instead of saying the code looks good.

None of that is hard. All of it is an extra step, and that is exactly why it does not happen. It is integration work with no demo, no launch moment and no clear owner. Model improvements arrive on their own, in a changelog, for free. The wiring has to be chosen by someone, and the person who would choose it is busy shipping.

So we end up with a review pipeline that is excellent at the cheap half of the job and structurally blind to the expensive half, and a team that has stopped noticing the difference.

What I changed

I am not going to hand you a ten step playbook. Two things were enough for me.

Before I open a diff, I write one sentence describing what the ticket asked for. Not what I built. What was asked. If I cannot produce that sentence without going back to the ticket, then I have not read the ticket, and that is now the signal I trust more than any checkmark. It would have caught mine in about fifteen seconds.

And the review bot now has to show its work on intent. If a PR has acceptance criteria, it walks through them one at a time. If it does not, it says so, loudly, instead of producing a thumbs up. A reviewer that cannot see the requirement has one honest thing to say about correctness, which is that it does not know.

The uncomfortable version of my story is not that I misread a criterion. It is that nothing in my workflow required me to read it, and automated review had removed the friction that might have made me. It nearly shipped. A person stopped it, and I would rather not run a process that depends on that. Connecting the ticket to the PR is boring, unglamorous integration work. Right now it is worth more than a better model.