WFTDA 2015 Playoffs Exstataganza!

Our massive look at the numbers behind the WFTDA postseason. Scoring and penalty data. official review trends, and a new stat to tell how good games really are.

…Continued from Page 1

Penalties

Here’s the breakdown of all the penalties committed in the 2015 WFTDA Division 1 playoffs, compared against 2014. You won’t be surprised by one of the things on this chart. You may be pleasantly surprised by the one at the bottom.

wftda-playoffs-penalties-d1-2014-2015

Finally! For the first time since we bothered to track stuff this in 2011, total penalties in Division 1 playoffs have gone down from the previous year. Though it’s a small decrease, an average of two penalties less per game across the whole 80 game season, just the fact that it isn’t an increase is significant. And it’s a minor miracle for a few reasons.

The WFTDA did not get off to a good start in Tucson, where the first Division 1 tournament had an average of 83.9 penalties and 1.3 foul-outs per game. A par weekend in Dallas brought the penalty average down to about 79 per, but halfway through the playoffs the WFTDA was still on track for another year of penalty increases.

But things settled down remarkably fast. In Jacksonville, the 17-game tournament averaged 66.6 penalties per game to reign the overall average back to nominal levels. After another typical weekend in Omaha, the best teams and best players in the world skated cleaner-than-average, averaging only 69 penalties per game at WFTDA Championships.

Adding it all up and dividing it by 80 games, the top 40 WFTDA teams — being officiated by the top referee crews in the WFTDA — managed to commit an average of 73.5 penalties per game in 2015. That’s lower than the 75.4 seen per game in 2014.

The decrease is awesome news, but let’s remember to put it into context. Six years ago, it wasn’t uncommon for the majority of competitive and/or high-level WFTDA games to finish with a penalty count in the low 60s, 50s, 40s, or even or lower. As the derby ground to a halt and became overly passive and defensive, penalties shot way up. Even the switch to 30 second penalties in 2014 didn’t slow down the rate at which skaters were getting sent to the box. So we’re still far away from where things used to be.

Plus, the good news of an overall dip in Division 1 penalties comes with some bad news. A lot of bad news, actually.

The rules the WFTDA has in place to deter dangerous contact don’t seem to be helping reduce dangerous contact. Collectively, back block, low block, and high block penalties went up more than 9% from the 2014 season in Division 1. If you compare it to 2013, the 2015 numbers represent an increase of 35%. Not good!

And then there’s forearms. All of the forearms.

Forearm fouls went from the fifth-most committed penalty in 2014 to the most common one in 2015. At a 23.4% increase, it was the single biggest jump in a penalty category on the year. And what a jump!

Helpfully, that increase was offset by decreases in direction of play penalties, track cuts, multiplayer blocks, and out of play calls. An armchair assessment of the games we saw during the playoffs and at championships might partially attribute these decreases to the faster pack speeds that were present in quite a few more games than in 2014. If you give players more room to move and more time to react, they’ll move and react illegally less often.

But what to attribute the increase in forearm penalties? That’s all on the WFTDA.

The 2015 rules update took the bold step of moving forearms beyond the remnants of the pre-2013 minor penalty system. Problem was, minor-impact forearms were so prevalent1, and skaters were so used them, the transition to the new rules was destined to be very rough.

Not helping matters was the new language the WFTDA used to redefine forearm penalties. Right from the drop of the hat, we identified issues in the 2015 rules that would potentially make forearm calls inconsistent. During the season, we checked in on some Champs-level play and found that forearm penalties were indeed being committed at an unusually high rate, even by some of the best players in the world.

That there was a big increase during the Division 1 playoffs should have come as no surprise. The real surprise is what happened in Division 2.

Warning: Those with a weak constitution may want to skip this next chart.

wftda-playoffs-penalties-d2-2014-2015

Forearm penalties: Up more than 50%. Direction of play penalties: Up nearly 50%. Total penalties in Division 2: Up 12.6%, the largest year-on-year increase we’ve ever seen in either WFTDA playoff division.

The crazy thing about this is that Division 2 had fewer penalties per game in 2014 than did Division 1. Not only were the D2 games more competitive overall, they were also cleaner overall, which was extremely impressive at the time.

In 2015, the D1 field cleaned up their act in many areas, enough to make up for the increase in forearm penalties and see an overall decrease. There were no similar improvements in Division 2.

As for the increases? There were so many penalties committed in D2, there was an average — average! — of 2.0 foul outs per game. Thirteen games had 90 or more total penalties. Four games had more than 100 penalties. One game, Kansas City vs. Houston, had 112 penalties.

One-hundred twelve penalties.

Another cringe-worthy penalty stat comes from the 2015 MRDA Championships. We tracked the whistles there, too, though with a different tournament setup (more teams and a weird consolation format) we’re not comfortable making a direct year-to-year comparison like with the WFTDA playoff seasons.2

But by itself, the breakdown reveals that 1.1 out of every 5 penalties committed in the men’s premiere tournament was the same type of penalty.

You’ll never guess which one!

mrda-playoffs-penalties-2015

Everyone watching the playoffs knew forearm penalties were a problem, but this big of a problem? Yikes.

So what happened? Did skaters not understand the rules changes (that they implemented)? Were refs not correctly enforcing the changes (that the skaters implemented)? Did skaters put too much spinach in their protein shakes?

Regardless of why, the WFTDA has already decided to stick to its guns, quietly announcing last month that there will be no rules update this year. The 2015 rules — forearm section and all — will continue to be in use through the 2016 playoff season.

Perhaps the extra year for skaters and refs to get used to the rule changes will help bring forearm penalties down to more reasonable levels? It’s a possibility. But we’d be more interested in seeing whether or not this problem, and those that may arise in future rules updates, disproportionately affects play in Division 2 and in the MRDA.

It’s fantastic that overall penalties are down in the premiere class. But the same rule set that Division 1 teams and referees use has got to work just as well for the (relatively) less-skilled skaters and referees in Division 2, the bigger-bodied skaters and equally-skilled referees in the MRDA, and the (absolutely) less-skilled skaters and referees that make up the vast majority of derbyfolk that play by WFTDA rules — including the ones that don’t have a vote in the WFTDA.

If the best players in the world are doing better, but everyone else is doing worse, is that really progress?

Later this year, we’ll get to see whether or not the rest of the WFTDA collective will see the penalty improvements trickle down from Division 1 after another year of digesting the 2015 rule changes and another season of tracking penalty calls.

Reviewing Official Reviews, 2015 Edition

After the 2014 playoff season, we did something that no one else in roller derby did, delivering stats on the success rate and trends of official reviews called for during the WFTDA playoffs. We learned some pretty damn interesting things last year.

In 2014 Division 1 playoff season, we counted 148 official reviews taken, but only 15 reviews resulting in the asking team getting their request granted. That made for an overturn rate of 10.2%. We also found that the amount of time wasted in unnecessary official reviews was like adding the equivalent of eight games of nothing happening on top of the 80 games of roller derby that did.

The official review process went through some changes in 2015, specifically allowing teams to use an official review as a time out without using a team time out. This had the effect of reducing the total number of official reviews in 2015, as well as legitimizing those that were requested. In 2014, the hit/miss ratio included some of the more speculative official reviews, which we can filter out for 2015.

Reviews used as time outs were clearly indicated by the WFTDA production team, both on the scoreboard and through the announcers, who were also fantastic in relaying the reason for the review request. They got pretty much all of them, which we’re super-grateful for, allowing us to collect them all and tell you just how many official reviews were ultimately necessary.

Our unofficial count is below, which disregards ORs used as TOs and official reviews that appeared to be official reviews but were not actually recorded as such.3 Not only can we tell you how many requests of each type were brought forward by Division 1 teams, we can also share how many of those requests were to try send an opponent to the penalty box or were in regards to the actions of one of the team’s jammers.

wftda-playoffs-official-reviews-d1-2015

The official review success rate in 2014 was horrible. In 2015, it was downright anemic. WFTDA Division 1 teams found that the requests they were throwing out were only sticking to the wall half as often as the year before, with just eight reviews being retained out of 137 used across 80 games, a success rate of just 5.8%.

This is very poor. And believe it or not, it might even be an optimistic percentage.

The sole success in the insubordination category was really more of a half-success. Late in the first half of their game at Champs, Gotham called for an official review against the Angel City bench for allegedly impeding a Gotham skater exiting the penalty box.

After a delay that lasted almost six minutes, Gotham finally convinced the officials to call a warning — not a penalty, just a warning — on Angel City. Because they specifically asked that a warning be given, Gotham kept their review. It went into our spreadsheet as such, but with a tart footnote:

Was that really worth all the trouble?

The high number of official reviews being called for in the WFTDA playoffs wouldn’t be bothersome if there were signs that these reviews were necessary to overturn bad calls or catch more missed calls. If this were happening, the retain rate for reviews would be increasing.

But it is not. Instead, teams continue to fish for penalties that surely they even know have a snowball’s chance in hell to go through, wasting the equivalent of seven full games’ worth of skating time over the course of the D1 playoff season.

As we pointed out last year, the reason why so many official reviews are dead on arrival is because of how the penalty enforcement system protects skaters. If referees are not 100% sure that an action warrants a penalty, the rules require them to not call a penalty.

During the 2015 Division 1 playoff season, teams used 75.7% of their official review requests to ask for a penalty on an opposing skater. The refs were right to not call a penalty in 97 out of 103 tries. Of these requests, the opposing jammer was targeted 85.2% of the time. Again, the refs knew what they were doing in 82 out of 87 instances.

Referees are humans, and there are of course times when they make mistakes. But they are not mistake-prone to the point where they need to explain to teams this frequently why calls and non-calls were correct to begin with.

For example, we continue to find it astonishing that bench coaches, which only have a narrow and often obstucted view of the action, so often insist that they saw a cutting penalty (or any penalty, really) that the crew of seven well-positioned, well-trained officials somehow didn’t. That cutting reviews are both the most requested type (40 out of 137; 29.2% of ORs) and overwhelmingly the least successful (2.5% reviews retained) in Division 1 tells you all you need to know about how silly the official review system can be sometimes.

As we did last time around, we strongly advocate for a change to the WFTDA official review system that requires a team requesting an official review to put one of their time outs up for stake. If the call stands, you lose the time out.

If the underlying intention of a team calling a review is to stop the clock and take a break, they should be forced to use a team time out for that. Reviews, and game delays they cause, should only happen when a team has a legitimate, non-speculative beef with a call.

This happened more often in Division 2, where the official review overturn rate was 14.3% after 11 of 77 review requests were successful. (This is still pretty bad, but definitely better than in Division 1.) There was a particular D2 game that explains exactly why a change to cut the fat out of official reviews wouldn’t adversely affect their necessity.

Against Bear City in Detroit, Wasatch found that they had to use all four of their potential allotted official reviews to pick up on calls the officials missed. In the first half, Wasatch went 1-for-2 on review requests, and in the second half they hit both of them, finishing the day with a highly unusual 75% success rate.

The coaching staff was clearly confident that these requests would come back in their favor, so any risk of a potential loss of timeout wouldn’t have been a worry to them had such a system been in place. Nothing would have been so different in the end.

However, for the 94.2% of requests in Division 1 which were not successful, coaches would suddenly have to be held to the same standard as referees: If you’re not 100% sure, don’t call it. If you force a team to risk losing a time out, but a team values their time outs, they won’t call for a reviews as often, especially those that only have a 5.6% chance of being successful.

That saves everyone the trouble of going through the motions just for the sake of doing so, keeping the focus on playing roller derby. It’s just common sense.

PPJ and The Statistical Best Games of the Playoffs

To wrap up our playoff exstataganza, let’s ask a simple question. What were the outright best games of the playoffs?

If you want to answer this question with your heart, as derby people often do, how can you not say Rose City beating Gotham in the last game of the season? With everything the team had been through, including the shocking and tragic passing of Rose City coach Rob Lobster, the way the they banded together and bamboozled Gotham’s defense with strong team offense was the perfect way to send off modern roller derby’s maiden broadcast on ESPN3.4

However, from a statistical standpoint it wasn’t the best game. It wasn’t even in the top three of Division 1, if you can believe that. One of the reasons? An above-average points per jam (PPJ) value of 9.78, coming from the result of several lopsided and uncompetitive jams in the game. Jam scores of 20-0, 20-0, 28-8  29-4, and 29-0 are what you’d expect to see in blowouts, not championship contests.

Our PPJ statistic is a measurement of how many average points are scored in a jam, for one game or across several games. The lower the PPJ, the more competitive any one jam is.

In basic terms, all this is saying is that a 4-0 jam (4.00 PPJ) is more competitive than a 9-0 jam (9.00 PPJ) for obvious reasons. It can also mean that two 4-0 jams (4.00 PPJ) are more competitive than one 4-4 jam (8.00 PPJ), since teams must compete against each other more often on initial passes for the same number of points.

Though it will fluctuate from game to game, in general a lower overall PPJ average is better for competition in the long run. If it’s hard for a team to score a lot of points in a jam, it would be even harder for them to score a lot more points than their opponent over several jams. When the equality of the teams are reflected by the equality of jam scoring, the true difference between two teams is more likely to be accurately reflected on the scoreboard throughout.

The PPJ average for all Division 1 games played in 2015 was 8.97. In Division 2, it was 8.23, which is also about what it was in Division 1 in 2014. Over the past three years, PPJ in the WFTDA has been holding steady in the 8.20 to 9.20 range, so small differences like this between the divisions aren’t very significant, statistically.

However, when we were counting up the OMG!!! games on the MJD charts from the previous page, we noticed a trend. Remember, these were the games that were not only close at the end, but also very close through at least half of the game. It turns out that the majority of these games were much closer than average on a per-jam basis, too.

Of the 13 OMG!!! games that happened in the WFTDA across Division 1 and Division 2 in 2015, nine of them had PPJ values that were typically two or three points lower than the overall PPJ average during the playoffs, usually in the 6.00~7.00 range.

Of the eight Division 2 games that averaged under 7.00 PPJ, seven of them had an MJD of 24 points or less, five of them finished with final score gaps of 20 points or less, and four them were OMG!!! games.

There’s quite a bit of competitive roller derby happening when PPJ is low.5 If we’re going to identify the best games of the playoffs, we’re not only going to want to find the games that are close at the end (low final score gap) and close after a majority of jams (low MJD), but also close during every jam (low PPJ). Games with a low penalty count, too, since we can’t very well have good roller derby when players are spending a lot of time sitting in the penalty box.

So more than just having games end close, we need to have clean skating teams engaging in constant competition against each other, resulting in a tight contest from start to finish. That’s roller derby at its best.

Despite the high rate of penalties across in Division 2, there were some games that were good in every major statistical category. Here are the top three we’ve identified.

Santa Cruz 154, Houston 141

  • 13 point Final Score Margin (8th closest in Division 2)
  • 8 point Median Jam Differential (T-4th tightest)
  • 6.86 Points Per Jam (T-6th lowest)
  • 71 Penalties (T-6th cleanest)

Houston, the 6-seed in Cleveland, shocked everyone with an upset win over 3-seed Columbia. After losing to Kansas City in the second round, they’d face Santa Cruz in the 3rd place game, the tournament 1-seed.

This game was all about the first half. The teams would trade the lead eight times, and never be more than 10 points apart up until halftime. Though Santa Cruz would take control in the second and never relinquish the lead, it was a lead that was barely more than 20 points for almost all of the rest of the jams.

Of course, a score gap that “big” is all relative. There were closer games, but no one was complaining during this one. And you can’t not appreciate the white-knuckle derby that was played to open things up.

Jet City 134, Columbia 133

  • 1 point Final Score Margin (T-1st closest)
  • 12 point Median Jam Differential (10th tightest)
  • 5.80 Points Per Jam (2nd lowest)
  • 71 Penalties (T-6th cleanest)

Up until the final jam, this game was pretty much everything you’d ever hope to see in roller derby. Tight defenses being thwarted by bold jamming and inventive offenses. This led to many one-pass and scoreless jams, though Columbia was getting more of them and as a result had the lead for most of the game.

Jet City then mounted a sharp comeback late, erasing a 29-point deficit over six jams (4.83 PPJ) to take a one-point lead. After quick 0-0 jam that saw the Columbia jammer head to the penalty box, Jet City lined up for the last jam with a power start.

What happened next was…quite unfortunate.

Right after the start of the last jam, Jet City’s blockers deliberately destroyed the pack, and kept it destroyed, giving the Columbia blockers no legal way to prevent the Jet jammer from getting lead status and ending the game. Another 0-0 jam, but one arrived at by way of a years-old rules loophole instead of by hard and fair competition between the teams.

It was a great game statistically, but one marred by an all too-familiar ending.

Chicago Outfit 141, Emerald City 121

  • 20 point Final Score Margin (10th closest)
  • 7 point Median Jam Differential (T-2nd tightest)
  • 6.55 Points Per Jam (5th lowest)
  • 61 Penalties (Cleanest overall)

There were closer finishes, but our spreadsheets are saying that the Chicago Outfit/Emerald City game from Detroit was in fact the best game of the 2015 Division 2 playoff season.

It could have very well been one of the closest games, actually. After the penultimate jam, Chicago Outfit only held a 6-point lead. In a game where the average jam saw 6.55 points, it was literally all to play for in the last jam. Emerald City’s jammer would seemingly get out first for lead…only to pick up a track cut penalty. The Outfit would coast to a 14-0 jam win and a 20-point victory.

What makes this win extremely impressive for Chicago is how clean they skated, committing only 23 penalties. Just as impressively, they did not commit a penalty in 23 of the game’s 40 jams! It was the lowest single-game penalty total for any one team in the entire Division 2 season by quite a bit, which is extra-amazing considering the big jump in penalties overall. Many congratulations to them for that performance.

Next, here are the top three statistical champions from the 2015 Division 1 season.

Gotham 143, Victoria 141

  • 2 point Final Score Margin (Closest overall in Division 1)
  • 10 point Median Jam Differential (6th tightest)
  • 7.47 Points Per Jam (18th lowest)
  • 67 Penalties (9th cleanest in competitive D1 games6)

Here’s a case where the closest game of the playoffs is also one of the best games of the playoffs overall. Though it could have been cleaner or more competitive on each jam, the closeness of the game throughout and at the end makes up for that nicely.

Victoria was leading the game from the start, and every time Gotham got really close, Victoria would pull away again. With 10 minutes to go, the Australians enjoyed a 25-point lead, the biggest of the night. With the VRDL defense playing the way it had been, that was a margin big enough for you to start wondering, for real, if Gotham would finally lose.

But no. Gotham milked every second out of the game clock they could, grinding out 36 points over the next 5 jams — 7.20 PPJ over that stretch, the going rate throughout the game. They survived a last-jam scramble by Victoria to hold on for a 2-point victory that was simultaneously improbable and inevitable.

Victoria 122, London 99

  • 23 point Final Score Margin (T-11th closest)
  • 20 point Median Jam Differential (T-18th tightest)
  • 6.31 Points Per Jam (3rd lowest)
  • 55 Penalties (2nd cleanest in competitive D1 games)

The first fully-international tilt at the WFTDA International Championships was a very grindy game, with long jams and punishing defensive walls. But it was a game capable of turning at a moment’s notice.

There was only one point in it after 10 jams. After this, Victoria would slowly build up to a lead of 20 points, keeping it there for more or less the entire game. (This explains the MJD of 20, which is quite high for a game this close.) Still, London was never out of it, needing one big jam to close the gap.

That big jam never came, however. After VRDL opened their initial lead, only once would either team get in more than two scoring passes during a jam. In 17 of these 24 jams, and 22 altogether, teams were held to one scoring pass or less. Rough!

The stats recognize this game as a top Division 1 game on the strength of per-jam competitiveness and the very low penalty count, despite not being the closest game in the world. In a way, it’s the opposite of the Gotham/Victoria tilt, which was close but not particularly clean or as competitive during jams.

Wouldn’t it be nice if we could take the best of both worlds? Have a game that’s not only close and super-competitive, but well-skated and light on penalties?

Well, it’s your lucky day. Such a game actually happened!

Jacksonville 149, Arch Rival 141

  • 8 point Final Score Margin (T-2nd closest)
  • 8 point Median Jam Differential (T-2nd tightest)
  • 6.17 Points Per Jam (2nd lowest)
  • 60 Penalties (6th cleanest in competitive D1 games)

There is absolutely no doubt about it: Jacksonville and Arch Rival played in what is simply the finest roller derby game seen in the WFTDA in a very long time. It was super-close, it was super-competitive, it was super-clean…and it was super-fun to watch!

Few anticipated a game like this happening. Jax had previously beaten Arch Rival by 81 points earlier in the 2015 regular season. While Arch was coming in to the playoffs having destroyed almost everyone else they had played, the thinking was if that if they played Jacksonville’s game — fast, fast, fast — they wouldn’t stand a chance in the rematch.

But Arch Rival did exactly that, matching Jacksonville’s jammers and blockers stride-for-stride. In 17 of the first 19 jams, including a stretch of 14 in a row, the teams could only trade single and partial scoring passes between one another. It was the same story in the second half, which saw cutthroat jam scoring in 13 straight jams. Altogether, neither team could muster more than 5 points in 36 of the game’s 47 jams.

What was happening to make the roller derby this competitive?

The answer came in a conversation between Lightning Slim and Captain Lou El Bammo, calling the game for the WFTDA.tv broadcast. After yet another 1-0 jam score during the 14-jam stretch of low jam scoring in the first half, a they remarked at what was happening in front of them.

Lightning Slim: “All that work for a single [point].”

Capt. Lou: “Yeah, there have been a lot of jams in this game where jammers have just had to call it off, to eke out a single point. … [It’s a] very low-scoring game. A combination of defense working it out very strongly on the track, and both jammers getting out and just having to hustle-hustle. Call off the jam, 0-point jams, 1-0 jams

When you have two strong defenses equally matched up with two strong offenses, it is inevitable that they will cancel each other out, allowing both teams to get their jammers out equally. (That is, within the same scoring pass.) If this happens a lot, you get a very low PPJ, representative of the extremely competitive roller derby happening.

During that first-half run of 14 close jams the announcers commented on, the PPJ was a mere 2.43. You only typically see scoring that competitive in forms of roller derby that allow the pivot to break and score without the need for a star pass. To see it happen in the WFTDA, which typically has relatively uncompetitive scoring in the 8-9 PPJ range, is nothing short of astonishing.

This still being the WFTDA, there were still quite a few jams with high scoring. The last three jams were 15-0, 12-0, and 8-6 (13.6 PPJ), which allowed Jacksonville to quickly erase a 21-point deficit, take a come-from-behind win over Arch Rival, and help raise the overall game PPJ to 6.17.7

That was still good enough to make it the second most-competitive contest, on a per-jam basis, during the 2015 Division 1 season. Add in how close the score was and how clean it was, and you’ve got a game that can make a fan out of anyone.

Indeed, a game like this should be a model for the WFTDA to look towards if it wants to improve gameplay and the quality of competition overall.

If one of the goals of the WFTDA is to foster improvements that result in better equality of competition, and fewer penalties to upset that competition, it should take a look at the games that best do those things. If the WFTDA can then help players emulate that kind of gameplay, either through strategy evolution for individual teams or via rules updates so everyone has equal access to it, the sport will only improve in the long term.

There have been many changes made by the WFTDA through the years, but in many aspects the numbers don’t completely support the idea that the organization has been moving in a wholly positive direction. Notably, improvements in one area (closer games in Division 2) have been getting washed out by steps back somewhere else (way more penalties in Division 2).

Still, it feels as if the sport on the verge of turning the corner in the places that matter the most. We’ll be keeping an eye on the stats to see, objectively, if the WFTDA can keeping following through on that.