Following the failure of the Senate to invoke cloture on the Halligan nomination yesterday and the Cordray nomination today, Steve Benen was a bit annoyed by a Politico piece that equivocates over which party is responsible for the rise in Senate obstruction over the last decades. Here’s Benen:

The Senate keeps an updated table, charting cloture votes by Congress over the last nine decades, using three metrics: (1) cloture motions filed (when the majority begins to end a filibuster); (2) votes on cloture (when the majority tries to end a filibuster); and (3) the number of times cloture was invoked (when the majority succeeds in ending a filibuster). By all three measures, obstructionism soared as Republican abused the rules like no party in American history.

Now, I’m not defending Politico here or saying that obstruction hasn’t increased over time — I think it has, and I think there’s some good evidence for it, especially in the case of nominations (which are the easiest to study, as it turns out, because there are no amendments and thus filibusters tend to be direct opposition, not strategic bargaining chips to win concessions). And I’m not going to wade into the debate over who or what party is responsible.

But proving that obstructionism is on the rise — or figuring out when it began or how much it has increased — is not as simple as Benen describes. Using cloture actions as a proxy for filibusters or obstructionism is a less than desirable metric, and also is open to easy political manipulation. Regardless of how many filibusters actually exist, the majority leader has the discretion (and often the strategic incentives) to produce very few or a whole lot of cloture votes.

Here’s a Venn diagram that will be helpful as I explain why. The entire box is the policy/nominations agenda, the largest circles are cloture motions filed and filibusters, respectively. The medium-sized nested circle is cloture votes taken, and the smaller nested circle is cloture motions invoked.

What we want to know is something like this: in any given Congress, what percentage of the large box (i.e. the total policy/nomination agenda) does the filibuster circle take up? And has that percentage been growing over time? What we know, for sure, is the absolute value of (1) cloture motions filed; (2) cloture votes taken; and (3) how often cloture is/isn’t invoked on those votes.

This leaves a bunch of problems:

1) Cloture motions are both under-inclusive and over-inclusive of filibusters. This is represented by areas A and B on the venn diagram. Area A includes times when a filibuster is occurring but the majority doesn’t file for cloture. This happens all the time: the majority informs the minority that they are thinking about bringing something to the floor, the minority says they will not give unanimous consent, and the majority just discards the idea and never brings it to the floor. Similarly, the majority might bring something to the floor, fail to get UC to move on it, and then just give up.  Area A also includes the situation in which a UC is reached that includes a 60-vote threshold for final passage. That’s obviously a filibuster, but there’s no cloture motion filed.

Area B is the opposite situation: a cloture motion filed on something that is not a filibuster. This occurs only occasionally, but it does happen: the majority leader thinks some floor action may be contentious, so he immediately files for cloture upon calling up a bill/nomination. It turns out there’s no filibuster, a UC agreement is reached for debate, and a final passage majority vote occurs with no cloture vote ever taken. But the converse happens all the time: no cloture motion is filed, a UC is worked out, and the bill gets a final passage vote. Whether or not there was a filibuster in either situation, or both, is unknown. But only one has a cloture motion associated with it. Similarly, the majority leader might occasionally pre-emptively file for cloture on a bill and then never take it up, because a different bill became the vehicle for the legislative action.

Now, none of this would be a huge problems if areas A and B could be precisely known or measured. But they can’t be. Even more to the point, the majority leader has almost complete control over the size of area A and area B. If he runs into a situation in which there’s a bill/nomination he would like to take up, but he knows that it will be filibustered and cannot pass, he then has the option of choosing whether to not take it up (area A), or going through the process of bringing it up, filing, and holding a cloture vote. That decision can be largely political, based on how the majority party thinks a failed cloture vote will play publicly. So depending on the political situation and temperament of the majority leader, an identical number of filibusters can be associated with a large number of cloture motions, or a small number. That is, to say, it’s not impossible that the observed increase in cloture actions over the last few decades is entirely the result of the majority leader taking a formerly private fight public. And while it’s unlikely that explains the entire rise in cloture actions, it’s almost certainly part of the equation.

One final point here: Area C, where there is a filibuster and cloture is filed, also contains a problem: more than one cloture motion can be filed on the same bill, and often is, in order to stack up potential cloture votes over a period of days. That means that any individual filibuster can have multiple cloture motions associated with it, again at the majority leader’s discretion.

2) The absolute number of cloture votes has the same problems, and another one. It seems pretty solid to think that a failed cloture vote (i.e. cloture not invoked) is evidence of a filibuster, as in Area D of the Venn diagram. But that’s not exactly true: if a cloture vote does not get even a bare majority of support (i.e. 50 or 51 votes), that’s good (but not airtight) evidence that it wasn’t a filibuster standing in the way of the bill/nomination, it was the preferences of the Senate, and is illustrated in teh diagram as Area E. This is not the case on many cloture votes, but it happens more often than you might imagine.

Still, there’s a bigger problem with trying to correlate total cloture votes (or failed cloture votes) with filibusters: any filibuster can be subject to multiple cloture votes. It seems silly to say that if there were 12 cloture votes on a single bill and all of them failed, that there were somehow 12 filibusters, rather that one. And once again, this puts the majority leader in the driver’s seat; if he’s stubborn and/or just wants to emphasize minority obstruction, he can pump up the number of failed cloture votes simply by holding cloture votes over and over again on the same measure. Which means, once again, that the political situation and the bargaining disposition of the majority leader is going to affect the number of cloture votes.

3) The absolute number of times cloture is invoked isn’t as good a measure as it sounds. For all the reasons already discussed, plus a few more. Area F is the standard case: the minority filibustered and the filibuster was broken by the majority getting to 60 votes. But the majority can get 60+ votes in situations when there really isn’t a filibuster (area G), or a situation in which a filibuster could have easily been defused without a cloture motion. For example, cloture can be used as a shortcut in many situations in which a UC was plausibly available. Say there are only a handful of objectors to a bill/nomination and both the majority and minority leadership are ready to move the bill. One option is to try to mollify the objectors, get a UC agreement to structure the debate, and go straight to final passage, which may get as many as 95+ votes. But the other options is to just file a cloture motion, do something else until it ripens, and then pick back up on it for the cloture vote.

This is especially attractive if the objectors have non-germane amendments they are trying to get into the mix. But the main point is this: there are occasionally cloture votes that pass by very large numbers, which often indicates that a UC might have been available and that the cloture vote was a strategic choice, not a necessity in response to filibuster. It also highlights the difference between a handful of objectors and a full-on minority party objection that has the support of he minority leadership. Both are technically filibusters and both are potential institutional problems, but they are very different things as far as the strategic options available to the majority are concerned.

4) The absolute value of the various cloture actions don’t control for the size of the policy agenda. This is perhaps the most important point. Even if we knew there was a perfect 1:1 correlation of cloture motions to filibusters, we still wouldn’t be able to say anything concrete about the use of the filibuster across Congersses, because saying “there were only 10 filibusters then but there are 15 now” doesn’t mean anything unless you know for sure there is a common denominator. If there were 30 things on the agenda “then” and 100 “now,” well, the filibuster rate — the percentage of the policy agenda subject to a supermajority threshold in the Senate — has actually gone down.

Now, the point in saying all of this isn’t to say that minority obstruction or use of the filibuster has not gone up over the past few Congresses. The point is that it’s a lot harder to judge than you might think, and it’s not at all clear that using metrics related to cloture are a good way to go about it. As described above, the majority leader has a lot of latitude in his strategic choices, and those choices can make cloture filings and votes go up or down, regardless of the number of underlying filibusters. And, also as mentioned, the absolute number of filibusters, even if it could be determined from the number of cloture votes, is not particularly interesting absent a sense of the size of the legislative agenda.

What I’ve said here isn’t a particularly new idea in political science. There are a number of scholars who have attempted to find alternative ways of measuring filibusters for quite some time. The best place to start if you are interested being Gregory Koger’s Filibustering, which uses news coverage to generate an independent measure of the number of filibusters and provides all the bibliographic references needed to locate others who have made similar attempts in the past. It’s not easy to count up filibusters, there’s definitely no perfect method. But methods like Koger’s are vastly superior to relying on cloture actions or other procedural count-ups in the Senate.

Update: Jon Bernstein has some related thoughts regarding the big picture of what’s going on in the Senate, and I definitely recommend reading it. I don’t think he disagrees with what I wrote (if I’m reading him correctly), and I more or less agree with him when he says:

I think that there’s a bottom line here that’s easy to overlook: the institutionalization of the 60 vote Senate. It’s not really a question, in other words, of whether any particular piece of legislation or nomination was attacked by filibuster; it’s that increasingly beginning in 1993, and overwhelmingly since 2009, minority parties have insisted that the majority produces 60 votes for everything.

It’s definitely my sense that minority obstruction has increased in the Senate; It wasn’t my intention to give off the impression that I think otherwise: it seems true on its face, and pretty much all research seems to confirm it. I’m just wary of using cloture actions as the measure; they purport to provide far too sure of an answer given how problematic the data can be.