It’s Manziel Time

Comments: No Comments
Published on: December 9, 2014

It’s Manziel time in Cleveland. I’ve discussed what I think of Cleveland in other posts. The short story is that I think the decisions of the Cleveland Browns front office resemble those a robot with a combination of dissociative identity disorder and schizophrenia. Once again we get to examine an important decision in Cleveland: the benching of Brian Hoyer in favor of Johnny Manziel. First, the facts.

Fact #1: Brian Hoyer has not been good this year

Not been good is putting it very mildly. Worst in the NFL is more correct. No quarterback has been allowed to be as inaccurate – 9.4% below league average – as Hoyer and still throw so many passes – 387 attempts. His inaccuracy is on par with Zach Mettenberger, Mike Glennon, Drew Stanton, and Ryan Mallett, all quarterbacks that began the year as back-ups and are unlikely to find permanent starting jobs any time soon. All in all, I don’t think much of Hoyer’s individual performance this year.

Fact #2: It is December and the Cleveland Browns have a winning record

At the time of this writing, Cleveland has a 7-6 record in the winningest division in football. By my quick reckoning, this has happened four other times in the last twenty years. Why does it matter if Brian Hoyer has been the worst quarterback in the league up to this point? You have won seven games with the worst quarterback, something must be working. Cleveland’s pass defense is coming up big week after week, they’ve got a group of receivers that showed they can spar with the best of them, and their running game has shown signs of life throughout the season. You got to seven wins doing something right. Why are you throwing that away now?

Fact #3: We don’t know what Manziel will do over three games

No one has any idea what Johnny Manziel will do during the final three games of the season. I have a prediction of Manziel’s quarterback rating of 76.6, but that’s over four years not three games. What is the purpose of upsetting everything that the Browns have built over the course of the season on the scant hope that Manziel can help you out? No one has any algorithm that says Manziel will be any better or worse than Hoyer over these final three games. And it is very possible that Manziel could be worse.

Fact #4: Coaching tenure in Cleveland

Perhaps the number that actually explains what is going on here. The average tenure over the last three head coaches in Cleveland has been a glorious 1.67 years. Mike Pettine likely doesn’t want to lower that average even further than it already is. He knows that sticking with Hoyer will likely get him a .500 team. Perhaps he sees the writing on the wall and knows that .500 won’t be enough to save his job. I don’t know that is the case, but I wouldn’t put it past the Cleveland management.

 photo blitzwing.jpg

Analytics in Washington

Comments: No Comments
Published on: December 2, 2014

Tony Kornheiser made some interesting statements on the radio the other day regarding what the organization in Washington D.C. should do about their terrible football team. You can read a more in depth piece about those comments here, but the main thrust of the situation was that Washington should start looking at analytics to improve their player selection process.

On the surface, I agree with this position. I am a firm believer in using useful mathematics to improve decision making processes. I started this blog in an attempt to inform people that we can develop individual player analytics in football and predict something about team performance from them.

However, in my humble opinion, Washington will not be the place that the football analytic revolution begins. Mostly I think that because of the actions of the team’s owner, Daniel Snyder. Before I say what I’m about to say, I want you to know that I have no intimate knowledge of Daniel Snyder. I’ve never even met the man. But I can observe his behavior and his public facing behavior leads me to believe something very important about Daniel Snyder and how he may see the world.

It is my impression that Daniel Snyder loves certainty. His behavior seems to follow a profile of X will do Y, Y will do Z, we want Z so therefore let’s go get X. He made a tremendous amount of money entirely on this principle. He had a small company that wanted to do something slightly different, he would go out and acquire another company that did that specific thing, and incorporate it into the original company’s machinery. He’s made his entire living on being able to understand the needs of his organization and then trusting that the assets he spends a tremendous amount of money to acquire will become worth more than what he originally paid for them.

Don’t get me wrong, Snyder’s certainty has served him well in the contexts he was in when he made his billions. Certainty can be a good thing for business leaders, mostly because it allows them to remain leaders. Humans are really bad at differentiating confidence from competence. Projecting certainty creates an environment where people will follow you. So, in some contexts, having lots of certainty makes a lot of sense. The only problem is, football is not one of those contexts.

It’s like my colleague who had difficulty driving on ice. She was originally from California and went to college in Arizona, so she never had to learn about what driving on icy roads is like. In addition, the person who taught her how to drive was a stock car driver. She was taught that to make the most efficient turns, you steer into the bottom of the turn, accelerate quickly through the bottom of the turn, and then steer out on the high side of the corner. And that works well for getting through turns quickly and efficiently. In addition, living in California and Arizona means you never have to confront the fundamental assumption that driving in such a way rests on – traction. If there’s no traction – like say when you’re driving on ice – one must drive in a completely different, opposite way. Accelerating through the bottom of a turn is a really good way to spin your wheels and wipe out. Instead, you have to always ensure that changes in direction never coincide with changes in speed. You can do one, but not the other. In my mind this is what Snyder is doing. He is taking a method that has always worked in the past, applying it to a different context, not recognizing that the underlying reality is different, and wiping out on the ice that is the process of building an NFL team.

You can see this in the future draft capital he gave away to move up to the #2 pick in the 2012 draft to get RG3. Generally speaking, it is a really bad idea to give away future draft picks to move up.  But such a strategy does make sense in a particular light, the light of certainty. If you absolutely feel like you know that this one particular player is going to work out, then it makes every bit of sense to act as Snyder did in 2012. Unfortunately, in football, having such certainty is disconnected from reality. The dirty little secret about player evaluation in football is that nobody knows who’s going to be good or bad. There are too many things to take into account. The amount of error in prediction is so astounding that no human brain can comprehend it. The best mathematical model I can create was accounting for about 15% of what makes a good NFL quarterback at last check. The reality of the NFL says that the way to create a winning team is to stockpile draft picks, evaluate everyone as if they were all drafted in the same round, and repeatedly draft multiple players at the same position (i.e. at least try out a new quarterback every single year).

The mindset you need to build a football team analytically is a mindset of uncertainty. You must accept the general premise that no one knows anything about anyone, the best models will get you 15% of the way to where you need to be, and you need to put yourself in a position to make luck work for you. I do not believe an individual like Snyder – a self-made billionaire used to projecting certainty from a leadership position – would value these qualities. There’s already the story of the economist that Washington hired to do analytic research for them in 2006 who quit after seven weeks of being marginalized in the organization.

The analytic revolution in football is coming quietly. The teams that end up doing analytics very well are not going to make a bit splash about it. The Seahawks and Packers come to mind as teams that, I believe, are on the forefront of the football analytics movement but are not saying a public word about it. Washington is simply not the place where people will be quiet about a new idea. And, ultimately, talking a new analytics department or bringing in some fresh-faced savior with their fancy mathematical model while demanding mechanistic links between actions and outcome will result in utter failure of the analytics process. If Washington brought someone like me into the organization, I feel like Snyder would demand I hit the gas at the bottom of the turn.   And while I might not be certain about much in football I am certain about this. Either I’d have to jump from the car or we’d both wipe out together.

Wide Receivers are a Pain to Evaluate: Part I

My original plan for a post this week was to talk about college football wide receivers and who I think is having the most productive season in college football right now.  But I ran into a problem.  The problem is that wide receivers are a giant pain to evaluate.  In fact, it gets incredibly frustrating to evaluate and project the performance of wide receivers.

Dependent Variables

The first question is what to use as an evaluation metric.  Before you can begin to predict “performance” in any useful way, one has to settle on what “performance” means.  Do you want yards, touchdowns, yards per reception, yards per target, touchdowns per target, fantasy points, what?

The question of dependent variable is crucial to understanding every analysis that comes afterward.  All the conclusions drawn from analyses done will only be relevant to the dependent variable that one chooses.  Therefore, it is crucial that one choose the “right” dependent variable.  Sadly, there is little consensus regarding what the right variable is to use to evaluate wide receivers.  So you’re stuck having to simply pick one.  I pick a witches-brewed version of yards per target. It works for me, but it may not work for you.

Depth of Target

Once you’ve “solved” your dependent variable problem, you run headlong into another one.  Generally, whatever DV you choose is somehow correlated with depth of target.  Wide receivers that get thrown to farther down the field rack up more yards, generally get more touchdowns (that one is a bit tenuous, but I digress), have more yards per reception, and more yards per target.  So now you’re stuck with a problem of understanding how the wide receiver fits into the offensive system regarding average depth per target.  Does this receiver have low yards per target because they are not a particularly good receiver or because they are consistently being asked to be a “chain-mover” out of the slot?  This calculation is impossible in some circumstances and tricky even with witches-brewed data.

Small Effect Size

You know what actually predicts production at receiver? Targets. End. This is a graphic I made showing the relationship between targets and yards in college football.

Targets accounts for around 75-80% of the variance in yards, which means that there isn’t much variance left for differences in ability to do any work.  You could have a pretty decent receiver buried on a roster and they won’t look like much at all *cough cough Jarius Wright cough cough*  And the reverse is also true.  A relatively poor receiver could get a lot of targets and look like a golden god.

So, I wanted to post about wide receivers.  I ended up getting frustrated at the position and writing about my frustration.

Quarterback Carousel

First off, happy Veterans Day, Grandpas. I miss you both.

Now to football. The quarterback carousel continues to spin. Everyone wants to know about how their team will do now that the new quarterback is under center. I’m going to look at three teams this week, the Eagles, Cardinals, and Texans and discuss the probable futures of each team.

Eagles

I am not worried one bit about the Eagles. I’m writing this after the Monday night where Sanchez went crazy, but don’t think I’m overreacting to one game here. Sanchez currently has a crazy high completion percentage for him. I’m fully expecting him to regress. Mark Sanchez has consistently been 4-6% below league average in terms of completion percentage depending on the year. We shouldn’t expect that he suddenly learned some profound bit of information about how to complete more passes. We should expect that he will return to his 4-6% below league average completion percentage over the course of the rest of the time he’s a starter in Philadelphia. But, you know who else was about 4-6% below league average completion percentage? Nick Foles. Honestly, I don’t see Sanchez being a detriment to the Eagles offense. I think Chip Kelly has a plan and that he is nothing if not adaptable. Philly will get through this leaning on their defense and their receivers.

Cardinals

The Cardinals are going to have more of a problem. The drop from Palmer to Stanton is going to be a much bigger drop than the drop from Foles to Sanchez. Stanton is consistently 3-4% poorer in completion percentage and he’s also not as effective throwing the ball down field. Palmer is generally league average in down field throws, but Stanton is more like bottom of the league in that category. Couple that with the fact that the Cardinals have really been punching above their weight up to this point in the season and you have a situation ripe for regression. The Cardinals should be very very worried.

Texans

The Houston Texans are the wild card. I did not see a quarterback switch coming for them. I’m not saying that Fitzpatrick is great. I know I predicted him to be a wildcard to lead the league in passing yards this year, but that prediction wasn’t as much based on him as it was everyone else around him. I believe I called him “serviceable” at the beginning of the year. And I still stand behind that assessment. He’s a little below average in completion percentage, but in an offense that’s more about throwing downfield than the average team we would expect that. Hopkins seems to be having a reasonable season and the team has already won twice as many games as it did last season.

Which is why I was very surprised to see that they’ll be going with Ryan Mallett for the foreseeable future. What exactly can Mallett offer you that Fitzpatrick can’t? Mallett has never started an NFL game and has thrown a total of 4 passes in an NFL game, one of which was intercepted. What can we even know about him?

Actually, we can know something about him since he played his college career recently enough to be part of my data set. I’ve even got a prediction about his career passer rating after four years in the banner up top (2011 draft class). The prediction in the banner is for a passer rating around 65. However, that prediction was based on a model I call “Mk. I” (Everyone seems to name their models and I’m an Iron Man fan). That model worked, but was based on Linear Regression and a data set that wasn’t as expertly cleaned as it could be.

Here’s what we learn about Ryan Mallett. I have a measure of college arm strength that helps differentiate quarterbacks. Mallett has the fourth highest score on that metric in a dataset that goes back to 2007. The three above him are Robert Griffin III, Andrew Luck, and Russell Wilson. However, arm strength is icing on the cake of an effective quarterback, but the cake itself. The cake of effective quarterbacking is accuracy and in that category, Mallett falls woefully short compared to the three other quarterbacks mentioned. When Mallett actually completes a pass, it goes for a long long ways. But he has tremendous trouble actually completing those passes. Basically, I see Mallett as Zach Mettenberger amplified. He’s got a cannon arm, but no ability to control it. The “Mk. III” model predicts his passer rating to be somewhere around 71. I think Fitzpatrick might be able to do a slight bit better.

What Exactly Do We Know Part III: The Nightmare Edition

Categories: Decision Making, NFL
Tags: No Tags
Comments: No Comments
Published on: October 28, 2014

This will be the final installment in my “What Exactly Do We Know” series. I think I’ve beat this horse enough that it’s about to fall over. But I need to talk about one final aspect of statistical reasoning and knowledge derived from data analysis. And this one is the creeping horror that should keep us all up at night. At the very least, this would keep me up at night if I were advising an actual NFL team. I’m going to being explaining the horror by having you imagine a job interview.

Actually, I’m going to ask you to imagine the lack of a job interview. How would you feel if you knew you were a top three candidate for a job, but the company called you one day saying “We don’t do interviews. We’ve already made our selection and we selected someone else because they had a higher college GPA.” When I ask my students to imagine this scenario, they say they’d be annoyed. They talk about how a particular score on a test doesn’t define them and if only they could get an interview they could prove their abilities and their worth.

However, if you’re trying to assess abilities and skills, evaluating on college GPA is actually the best way to get the skills and abilities you’re interested in. In fact, trying to assess abilities and skills with an unstructured conversation is one of the best ways to introduce unintended and significant bias into your decision making process. Most large, modern organizations don’t even use a conversation-style interview to assess skills anymore. Conversation-style interviews are done to only answer whether the person interviewing you could stand working with you for a day. But I digress.

I bring up job interviews because they are a fascinating point of the employee selection process. If used in the old let’s-chat-for-20-minutes way, the interviewer is unlikely to see the person’s worth with any form of accuracy. Which brings us back to football.

Football is an amazingly interesting game because of how interdependent all the action is. However, the interdependence leaves us with a fundamental problem. Can someone looking from outside the situation truly see what is actually happening in a quarterback-wide receiver connection? I looked in the published academic literature and couldn’t find the study that directly answers that question. I’m running the study in my lab right now, but I won’t have an answer for you for a long while. I haven’t looked at the data yet, but the tangentially related studies all seem to indicate that the answer is “No, we can’t see who is responsible for what when looking from the outside.” And, assuming I’m right, how then can we trust the opinion of any talent evaluator that doesn’t attempt to systematically control for such biases? Can even the most relevant talent evaluators, namely those that make personnel decisions for NFL teams, be trusted to make the right evaluation?

My top quarterback prospects from the 2014 draft were Nathan Scheelhaase and Keith Price. At the moment, neither of these players is on an NFL roster. The internet currently does not record what Scheelhaase is up to, but Prince is a quarterback – a backup for the Saskatchewan Roughriders in the Canadian Football League. So…what are we to make of this?

Keith Price 2013.jpg

Let’s say I’m right and we can’t trust talent evaluators that don’t use data to control the biases. This means that I can put out a list of prospect, those prospects can go out into the league and get evaluated. In the case of Price, he was evaluated by two of the best in the business – the Seahawks and Patriots.   Neither team desired his services which is how he ended up in Canada. But according to the theory we put out in the first paragraph, the fact that he didn’t get picked up doesn’t mean anything. We already believe that the evaluators can’t control a human bias. Hopefully you understand that this is a very advantageous rhetorical position to be in. How can you convince me that I’m wrong? What evidence would I accept if I won’t accept the pre-season evaluation of two teams who I have already stated I believe a two of the best in the league in evaluating talent?

The only evidence that the model will currently accept is on-field, regular season outcomes. And not only that, but I would need a lot of attempts to actually consider the notion I’m wrong. Which is why I would lose sleep if I worked for an NFL team. I’d have to trust the wings of Bayesian statistics to a degree I’ve never had to in the past. How terrifying would that be?

What Exactly Do You Know, Part II: The “Is He Your Cousin or Something?” Edition

Comments: No Comments
Published on: October 21, 2014

In last week’s post, I discussed the concept of what statistical inference actually tells you and how it’s boring and cumbersome to talk about it accurately, so analysts often shorten the conversation so they can actually talk with real people about something interesting. Today we take a slightly different tack regarding what exactly we know. Our example for this week is Minnesota Vikings quarterback Christian Ponder.

Christian Ponder close-up.jpg

If you’ve been reading for a while, you know that I was actually a fan of Ponder for a long time. Or, at the very least, I didn’t hate him with every fiber of my being like every other Vikings fan seemed to. I called him “not the problem in Minnesota” instead pointing to the largely ineffective receiving corps. I was talking to my neighbor before the season started. I said that Ponder is not the problem. We had this long and somewhat loud conversation about how I have to be wrong about him because everyone was giving up on Christian Ponder. Even Paul Allen – the radio play-by-play announcer for the Vikings – a guy who has never in his life given up on anyone in a purple jersey had given up on Christian Ponder. When I persisted that Ponder wasn’t the problem, my neighbor ended the conversation by saying, “You’re the only guy I know saying nice things about Ponder. Is he your cousin or something?” At the time the comment made me laugh. Then the Thursday night game against the Packers happened. I had to think more about this and examine what I know and what I don’t know about Christian Ponder in particular and the game of football in general.

So why was I so adamant that Ponder wasn’t the problem? Because, for all his faults, Ponder has one singular but important ability. He is rather accurate for an NFL quarterback. He’s not super-star Peyton Manning accurate, but he can get a football into a receiver’s hands slightly better than the average NFL quarterback. And why do I care so much about accuracy and nothing else? Because it’s the only quarterback ability I’ve found at the NFL level that will predict useful outcomes. Nothing else comes back predictive. Not a quantification of arm-strength, not Wonderlich scores, nothing at the combine, nothing but accuracy predicts NFL level outcomes.

And now we have another trap that analysts can fall into, a trap that is particularly present and meaningful for the NFL. I can’t find a predictive effect of my in-house metric that I think measures arm strength (let’s ignore the measurement point of “how do we know this thing is really arm strength” for now. It’s important but not where we’re going here). So I don’t find this effect. There are a couple possibilities why. The first possibility is the one that brings the page views and the loud conversations – that Arm Strength isn’t an important thing. However, another interpretation is that the lack of data at the NFL level makes finding the effect of arm strength insanely difficult.

Think about it like this. Imagine I told you that there was gold to be found in the body of water closest to you. To me that body of water is a river, so for the rest of this example I’ll be talking about a river. But maybe for you it’s a lake or an ocean or your friend’s bathtub. Whatever. You want to find this gold because you think having gold would be better than not having gold. So you go out and buy all the equipment necessary to pan for gold. You get the sorter pieces and the dirt sucker and everything else and you go stand in the river for a few hours and try to find this gold. Now, if you stood in the same spot panning for gold for four hours and didn’t find gold, would it be reasonable for anyone to assume that I’m wrong and that there is no gold in the river?

 

Crude Drawing of Where Gold is in a Fictional River
Crude Drawing of Where Gold is in a Fictional River

No, it would be ridiculous to say that. Maybe you were panning in the wrong spot. Maybe the screen you were using was too big and all the gold was little and slipping through. There could be many reasons why you didn’t find gold in the river.

Analytical findings are like gold. Just because you don’t find one, doesn’t mean that they aren’t there. This is a concept called “statistical power” and in the NFL it’s a huge problem. Our ability to find effects generally increases the more data we have. Think of it like this – more data makes our gold panning screens smaller. It allows us to find ever smaller nuggets of gold. In the NFL, the data is very sparse. There are only 32 teams playing 16 games each with maybe 30 passing attempts in each game. This pales in comparison to basketball’s 82 games and baseball’s 162. Compared to other sports, an effect in the NFL has to be fairly large before our screens will catch it. There is so little data coming from the NFL that it’s possible an arm-strength effect exists but there just isn’t enough data to find it.

So, after the Thursday night Ponder debacle, I went on a quest for more power. And in football, if you want more statistical power you need to look at the college level. With many many more teams we suddenly have a lot more power in our data set. I spent most of my summer calculating the same arm-strength metric for every NCAA FBS level quarterback and I ran the same model to see if arm-strength, along with accuracy, can predict useful quarterback outcomes. Low and behold, it does (said the amazed analyst and no one else). Ponder fairs very well on accuracy, but he suffers horribly on arm-strength. With this lesson learned, it’s time to quit dying trying to take the Ponder hill. Ponder is a problem for the Vikings offense. One of many, many problems.

What Exactly Do You Know?

Categories: General Info, Statistics
Comments: 1 Comment
Published on: October 14, 2014

(This post was inspired, in part, by a post Matt Waldman posted on his website a few days ago. If you haven’t already, go check out his site. He does an amazing job over there. The post in question is titled “Deny Emotion and You Only See a Fraction of the Game.”)

The establishment in sports has, for a while now, been telling outsider-sports-analytics types that one of the main barriers to widespread acceptance of analytics rests on the ability of the quants to communicate with the non-quants in ways that the non-quants can understand. I’ve covered the problems of communicating information both to quants and to non-quants in the past. But Matt’s post about emotion made me realize something more.

His general point was about the role emotion plays in sports performance. Specifically, he was contemplating whether or not momentum exists in sports. He cites conversations with analytics experts on the subject of momentum. If you’re familiar with the argument, most analytics experts conclude momentum does not exist. This is a fairly standard finding across many sports – basketball’s hot hand being another classic example. Matt’s reaction to that conclusion is also fairly standard among non-analysts. His argument is that emotion, and the effects of emotion on situations in sports are obviously occurring. Anyone who denies that is missing a huge portion of the signal inherent in the game. At one point he suggests that analysts have possibly never put themselves in physically dangerous situations and felt the impact of emotion first hand.

So, two things with that summation, both having to do with communication. First, assuming experiences that someone has or hasn’t had is a raw nerve for me. The stereotype of the milquetoast academic who simulates experiences rather than having actual experiences looms large over me. I’m guessing it has something to do with my father telling me to put down the video games and go spend some time working on my grandfather’s farm. But, I know the comment wasn’t directed at me and it wasn’t malicious anyway, so let’s put aside any irritation that might shut our brains off. In fact, we need to keep our brains on if we’re going to truly examine the point Matt is trying to get at.

Analysts (myself included) are often guilty of a particular linguistic shorthand. Our job is to find predictive effects. Does changing the way a request is phrased reliably change donations to charity? If I know about your height, can I make an educated guess about your weight? That sort of thing. We can become so familiar, so practiced in that job that, when we talk to other people, we tend to shorten the description of what we’re talking about. When my neighbor asks me what I know about momentum in sports, I say “I’m trying to find out if momentum exists” and he gets excited and engages with the conversation. The problem for analysts, though, is that this gets the conversation off on a disingenuous foot. Because we’re not truly trying to find out if momentum exists. We’re trying to find out if the predictive effect of momentum on some other variable, like scoring, exists. But we repeat the phrasing about studying the existence of momentum so much that we can forget that other people see that collection of words as having a different meaning.

And when we analyze the results in sports, the results a pretty clear. The effect of momentum on any variable we look at is unpredictable. The same is true in the academic literature as well. We cannot predict the effect of emotion on motivation with any reliability. So, I agree that we should probably stop saying that momentum doesn’t exist. The subjective experience of the emotion of the game is a real thing that people feel.  But I will hold fast to the notion that predicting what will initiate a change in momentum and how a change in momentum will impact athletic performance is an unpredictable enterprise.

NCAA Quarterbacks: 2015 Draft Class

The quarterback situation for the 2015 draft class is looking very murky.  The 2012 draft class was a very unique class in that the highly talented players all managed to get drafted highly and to generate the needed playing time to demonstrate that talent.  I don’t see that happening with the 2015 draft class currently.  I see the 2015 class looking a lot more like the 2013 draft class.  Guys that can play exist in the pool, but who knows if they will get the playing time they need.

Revisiting the quarterbacks I’m watching section from earlier this year, I’m still high on Rakeem Cato.  I saw one feature story on him, but it had to do more with his background than with his ability as a passer.  Hopefully he gets some more attention for the latter.

One player we should keep a close eye on is Bo Wallace at Ole Miss.  I didn’t have him on my original list this year, but he’s having a very good season, both in terms of accuracy and in down the field throws.  That second part is important because he did not have a good season throwing down field last year.  Good to see him demonstrate that he actually has that ability.

Last but not least, someone to keep an eye on is Conner Halliday at Washington State.  Not saying I’m overly excited about him, but he does have something worth looking at.  I don’t know where he’ll end the season in my projections, but his current performance is at least elevating him from where he was.

Last but not least, looking far into the future, is anybody looking at this sophomore from Middle Tennessee?  His name is Austin Grammer.  A name you might want to get to know.

Bills Bench E.J. Manuel: What’s the Plan?

Comments: No Comments
Published on: September 30, 2014

The Buffalo Bills have benched their starting quarterback and hope of the franchise, 2013 edition, E.J. Manuel. We don’t know if we’ll ever see him on a professional football field again. As of this writing, Manuel has a career passer rating of 78.5 and a career adjusted net yards per attempt (ANY/A) of 5.9. But this post isn’t about E.J. Manuel and what he has or hasn’t done on the football field.

My prediction of where E.J. Manuel would be after four years in the league was a passer rating of 74.8 with an ANY/A around 4.8. By my estimation he’s performed right about where I expected, maybe even a little better on ANY/A. But one case doesn’t prove the worth of a process and this post isn’t about me.

This post is about the Bills’ front office. I’m very curious what they expected of E.J. Manuel. In fact, I’m curious what they expected of their entire football team. Did they believe that, in 2012 when they went 6-10 that they were just a few players away from contending for a playoff spot? Did they truly believe that when they haven’t had a winner record since 2004?

I wonder what they expected on draft day in 2013. They drafted a new quarterback, certainly, but they’d also drafted a couple wide receivers in Robert Woods and Marquise Goodwin. I wonder if they considered how difficult it would be to evaluate the performance of a new quarterback when both quarterback and receivers were changing. I wonder how much hope existed in the draft room on that day and on what evidence that hope rested. I wonder if they felt the same way at the end of the 2014 draft when they had given away their 2015 first round pick to Cleveland to draft another new wide receiver in Sammy Watkins.

I wonder if, when making the decision to bench E.J. Manuel, the decision makers in the Bills organization thought about benching Robert Woods instead, a player I have currently rated as the 203rd best pass receiver in the NFL. I wonder if someone is thinking, “We threw the ball Wood’s way 12 times this week and gained a total of 17 yards on those 12 attempts. I wonder if the receiver had any impact at all on such a stat line.”

I wonder what the decision makers in Buffalo expect from Kyle Orton, a player whose career per attempt statistics look remarkably similar to the guy he’s replacing. What exactly do they think is going to happen here? We have a pretty good idea what Kyle Orton is going to do with a football in his hands. And we also know that Orton is old enough and experienced enough that he probably isn’t going to get much better. Manuel, whatever his faults might have, at least has a chance to improve.

So much in the sports world comes down to expectations. We expect E.J. Manuel to be good because he was the first quarterback taken in the 2013 draft, and the only quarterback taken in the first round. But why did we expect so much of E.J. Manuel? Because of how much someone else valued him? Because of what we thought we saw during his college days? Sadly, human minds cannot correctly weigh those expectations. Data is needed so that we don’t let our expectations interfere with our ability to make the best decision we can at this exact moment.

Mostly, Buffalo Bills decision makers, I wonder…

 

Ryan Tannehill – Using Passer Rating to Evaluate Quarterbacks

Categories: NFL, Statistics
Comments: No Comments
Published on: September 24, 2014

Apparently Ryan Tannehill doesn’t have a high enough passer rating this season. At least, not high enough for his coach, Joe Philbin to commit to him being the starting quarterback for the Dolphins on two separate occations. I first saw this yesterday and assumed it was some thinly veiled motivational tool. But then it happened again today. Which doesn’t happen if you’re trying to use shame as a motivational tool…at least, not if you’re doing it right.

So let’s assume that Philbin is seriously considering benching Ryan Tannehill and putting in Matt Moore because Tannehill’s passer rating isn’t high enough. What are the implications of such a move?

First, Ryan Tannehill has not been good this year. In fact, he’s been one of the worst quarterbacks in the league over the first three games of the season, depending how you figure it. Philbin isn’t wrong that Tannehill’s passer rating is awful. However, and this is the really important part, who cares?

Whenever you start using numerical information to inform real world decisions, you must be incredibly careful to select the correct criterion variable. Criterion variables are the regression equivalent of dependent variables, a variable we care about but don’t mess with. We just allow it to vary naturally. Proper selection of your criterion variable is one of the most important decision you make when beginning a program of analysis. Every truth you uncover during your studies will only be true with respect to the criterion variable you chose. That single decision will color every single decision from the moment you make it on.

Returning to the last question I asked, who cares if Tannehill’s passer rating isn’t particularly good? Most people in the football analytics community would say no one should care. Pro Football Focus loves saying that passer rating tells you about the efficiency of the offense, but not the individual running it. Others like Chase Stuart at Football Perspective talk about how the weights assigned to passer rating are arbitrary and incorrect. These heavy hitters in the analytics community say that Philbin is using the wrong criterion variable to evaluate his quarterback. And I would largely agree with these perspectives.

So what does Philbin really want out of his quarterback? That’s the ultimate question in football. What criterion variable should we choose to actually understand the game and the people who play it? Those of us interested in understanding the game of football from an analytic perspective should be scrambling trying to understand everything we can about what different criterion variables tell us about the game. The correct selection of criterion variables is one of the most critical questions we face at this moment.  We know passer rating isn’t particularly good. ANY/A and ESPN’s QBR certainly make better cases as being good criterion variables, but the perfect variable doesn’t exist.

Ultimately, my guess is that Philbin would just like his offense to contribute to a win. But, if Joe Philbin is really going to use passer rating to evaluate quarterbacks, I could have told him to avoid Tannehill in the first place.

«page 1 of 8

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Join 1 other subscriber


Welcome , today is Wednesday, December 17, 2014