During last night’s game between the Saints and Steelers, there were two plays challenged in the first half that I think illustrate well the fundamental irrationality in both the decision-making process for challenging among coaches, and the evaluation of the decision to challenge by the announcers. 

The fundamental irrationality is this: coaches seem to be trying to maximize the percentage of challenges they win, not the total benefits gained by challenging (and the announcers seem to be evaluating the same way). Whatever the probability of a challenge being successful, there is a second probability that should highly influence your decision to challenge or your evaluation of someone’s decision to challenge: the benefit accrued from the challenge, if  successful.

For instance, imagine that the other team has the ball on your 15-yard line and completes a pass to the one-inch line that is ruled a touchdown. The video evidence is clear that it is not a touchdown. Should you challenge?

Probably not. Because even if you are successful, all you reap is a situation in which the other team has 1st and goal from your 1-inch line. I don’t know offhand what the touchdown rate is for 1st and goal from within the 1, but I would guess it’s close to 90%, and the expected value of having the ball in that situation is probably more than 6.5 points. So all your challenge can possibly do it gain you about 1/2 point in expected value. This wouldn’t be a problem if you had unlimited, costless challenges. But you don’t: a bad challenge costs you a timeout; any challenge (good or bad) reduces your number of possible future challenges, some of which might be worth much more than half a point.

So any challenge strategy should have two components in the formula : the probability of success and the benefit of succeeding. A simple multiplication is good enough; the point is simply that a higher probability of success does not make a better challenge, nor does a high success rate make for a better strategy. You’d much rather make a challenge that only has a 10% chance of being right on a game-changing play than make one that is a lock to be correct but that hardly matters. After all, the goal is to maximize the point differential between you and your opponent, not the correctness of the officiating.

Of course, the exact situation above happened in the game last night: On 3rd and 4 from the New Orleans 12, Pittsburgh completed a pass to the goal-line. Ruled a touchdown, New Orleans challenged and won, putting the ball at about the six-inch line, first and goal. The announcers declared it an “easy challenge” because it was so obviously going to be overturned; not once did they assess the logic of wasting a challenge on something that is unlikely to change the ultimate outcome of the drive.

But then fuck all, what do you know, New Orleans stops Pittsburgh on the next three plays from the goal line and the Steelers have to kick a feel goal. And we head to timeout with the announcers crowing about how key the challenge was. Kill me.

The other important example happened last night on a punt. Pittsburgh punted to the New Orleans 30 or so, and the New Orleans return man got back to around mid-field, where he lost the ball. The ruling on the field was that he was down by contact and that the ground caused the fumble. The video evidence was pretty clear; it was highly probable that he was down and that the call was right, but it wasn’t certain. Nevertheless, Mike Tomlin — who has the lowest challenge rate in the league — threw the red flag. The announcers were largely incredulous. “I don’t really know what he’s thinking.”

I do. He’s thinking about the second half of the equation. Any challenge that changes who has possession of the ball is a highly consequential challenge (even more so if it occurs deep in either team’s territory). Obviously, midfield isn’t as great as either red zone, but I would think that even if this challenge has only a 20% chance of succeeding, it’s value (probably something like a 5 point swing overall, but I haven’t looked into it) is going to be higher than the New Orleans challenge on the goal-line questionable TD.

Of course, Tomlin lost the challenge and the announcers shook their heads.

So we had two situations last night that will reinforce the blind-spot on challenges. Coaches will continue to think that the proper metric is “percentage of challenges won,” which will encourage them to throw the flag in spots that aren’t of much value, and keep it in their pocket during high-value situations in which their probability of winning the challenge is low. Announcers will follow suit. And I will continue to seethe every time the Giants challenge touchdowns just to make it first and goal inside the five.