It’s Manziel Time

Comments: No Comments
Published on: December 9, 2014

It’s Manziel time in Cleveland. I’ve discussed what I think of Cleveland in other posts. The short story is that I think the decisions of the Cleveland Browns front office resemble those a robot with a combination of dissociative identity disorder and schizophrenia. Once again we get to examine an important decision in Cleveland: the benching of Brian Hoyer in favor of Johnny Manziel. First, the facts.

Fact #1: Brian Hoyer has not been good this year

Not been good is putting it very mildly. Worst in the NFL is more correct. No quarterback has been allowed to be as inaccurate – 9.4% below league average – as Hoyer and still throw so many passes – 387 attempts. His inaccuracy is on par with Zach Mettenberger, Mike Glennon, Drew Stanton, and Ryan Mallett, all quarterbacks that began the year as back-ups and are unlikely to find permanent starting jobs any time soon. All in all, I don’t think much of Hoyer’s individual performance this year.

Fact #2: It is December and the Cleveland Browns have a winning record

At the time of this writing, Cleveland has a 7-6 record in the winningest division in football. By my quick reckoning, this has happened four other times in the last twenty years. Why does it matter if Brian Hoyer has been the worst quarterback in the league up to this point? You have won seven games with the worst quarterback, something must be working. Cleveland’s pass defense is coming up big week after week, they’ve got a group of receivers that showed they can spar with the best of them, and their running game has shown signs of life throughout the season. You got to seven wins doing something right. Why are you throwing that away now?

Fact #3: We don’t know what Manziel will do over three games

No one has any idea what Johnny Manziel will do during the final three games of the season. I have a prediction of Manziel’s quarterback rating of 76.6, but that’s over four years not three games. What is the purpose of upsetting everything that the Browns have built over the course of the season on the scant hope that Manziel can help you out? No one has any algorithm that says Manziel will be any better or worse than Hoyer over these final three games. And it is very possible that Manziel could be worse.

Fact #4: Coaching tenure in Cleveland

Perhaps the number that actually explains what is going on here. The average tenure over the last three head coaches in Cleveland has been a glorious 1.67 years. Mike Pettine likely doesn’t want to lower that average even further than it already is. He knows that sticking with Hoyer will likely get him a .500 team. Perhaps he sees the writing on the wall and knows that .500 won’t be enough to save his job. I don’t know that is the case, but I wouldn’t put it past the Cleveland management.

 photo blitzwing.jpg

Analytics in Washington

Comments: No Comments
Published on: December 2, 2014

Tony Kornheiser made some interesting statements on the radio the other day regarding what the organization in Washington D.C. should do about their terrible football team. You can read a more in depth piece about those comments here, but the main thrust of the situation was that Washington should start looking at analytics to improve their player selection process.

On the surface, I agree with this position. I am a firm believer in using useful mathematics to improve decision making processes. I started this blog in an attempt to inform people that we can develop individual player analytics in football and predict something about team performance from them.

However, in my humble opinion, Washington will not be the place that the football analytic revolution begins. Mostly I think that because of the actions of the team’s owner, Daniel Snyder. Before I say what I’m about to say, I want you to know that I have no intimate knowledge of Daniel Snyder. I’ve never even met the man. But I can observe his behavior and his public facing behavior leads me to believe something very important about Daniel Snyder and how he may see the world.

It is my impression that Daniel Snyder loves certainty. His behavior seems to follow a profile of X will do Y, Y will do Z, we want Z so therefore let’s go get X. He made a tremendous amount of money entirely on this principle. He had a small company that wanted to do something slightly different, he would go out and acquire another company that did that specific thing, and incorporate it into the original company’s machinery. He’s made his entire living on being able to understand the needs of his organization and then trusting that the assets he spends a tremendous amount of money to acquire will become worth more than what he originally paid for them.

Don’t get me wrong, Snyder’s certainty has served him well in the contexts he was in when he made his billions. Certainty can be a good thing for business leaders, mostly because it allows them to remain leaders. Humans are really bad at differentiating confidence from competence. Projecting certainty creates an environment where people will follow you. So, in some contexts, having lots of certainty makes a lot of sense. The only problem is, football is not one of those contexts.

It’s like my colleague who had difficulty driving on ice. She was originally from California and went to college in Arizona, so she never had to learn about what driving on icy roads is like. In addition, the person who taught her how to drive was a stock car driver. She was taught that to make the most efficient turns, you steer into the bottom of the turn, accelerate quickly through the bottom of the turn, and then steer out on the high side of the corner. And that works well for getting through turns quickly and efficiently. In addition, living in California and Arizona means you never have to confront the fundamental assumption that driving in such a way rests on – traction. If there’s no traction – like say when you’re driving on ice – one must drive in a completely different, opposite way. Accelerating through the bottom of a turn is a really good way to spin your wheels and wipe out. Instead, you have to always ensure that changes in direction never coincide with changes in speed. You can do one, but not the other. In my mind this is what Snyder is doing. He is taking a method that has always worked in the past, applying it to a different context, not recognizing that the underlying reality is different, and wiping out on the ice that is the process of building an NFL team.

You can see this in the future draft capital he gave away to move up to the #2 pick in the 2012 draft to get RG3. Generally speaking, it is a really bad idea to give away future draft picks to move up.  But such a strategy does make sense in a particular light, the light of certainty. If you absolutely feel like you know that this one particular player is going to work out, then it makes every bit of sense to act as Snyder did in 2012. Unfortunately, in football, having such certainty is disconnected from reality. The dirty little secret about player evaluation in football is that nobody knows who’s going to be good or bad. There are too many things to take into account. The amount of error in prediction is so astounding that no human brain can comprehend it. The best mathematical model I can create was accounting for about 15% of what makes a good NFL quarterback at last check. The reality of the NFL says that the way to create a winning team is to stockpile draft picks, evaluate everyone as if they were all drafted in the same round, and repeatedly draft multiple players at the same position (i.e. at least try out a new quarterback every single year).

The mindset you need to build a football team analytically is a mindset of uncertainty. You must accept the general premise that no one knows anything about anyone, the best models will get you 15% of the way to where you need to be, and you need to put yourself in a position to make luck work for you. I do not believe an individual like Snyder – a self-made billionaire used to projecting certainty from a leadership position – would value these qualities. There’s already the story of the economist that Washington hired to do analytic research for them in 2006 who quit after seven weeks of being marginalized in the organization.

The analytic revolution in football is coming quietly. The teams that end up doing analytics very well are not going to make a bit splash about it. The Seahawks and Packers come to mind as teams that, I believe, are on the forefront of the football analytics movement but are not saying a public word about it. Washington is simply not the place where people will be quiet about a new idea. And, ultimately, talking a new analytics department or bringing in some fresh-faced savior with their fancy mathematical model while demanding mechanistic links between actions and outcome will result in utter failure of the analytics process. If Washington brought someone like me into the organization, I feel like Snyder would demand I hit the gas at the bottom of the turn.   And while I might not be certain about much in football I am certain about this. Either I’d have to jump from the car or we’d both wipe out together.

Wide Receivers are a Pain to Evaluate: Part I

My original plan for a post this week was to talk about college football wide receivers and who I think is having the most productive season in college football right now.  But I ran into a problem.  The problem is that wide receivers are a giant pain to evaluate.  In fact, it gets incredibly frustrating to evaluate and project the performance of wide receivers.

Dependent Variables

The first question is what to use as an evaluation metric.  Before you can begin to predict “performance” in any useful way, one has to settle on what “performance” means.  Do you want yards, touchdowns, yards per reception, yards per target, touchdowns per target, fantasy points, what?

The question of dependent variable is crucial to understanding every analysis that comes afterward.  All the conclusions drawn from analyses done will only be relevant to the dependent variable that one chooses.  Therefore, it is crucial that one choose the “right” dependent variable.  Sadly, there is little consensus regarding what the right variable is to use to evaluate wide receivers.  So you’re stuck having to simply pick one.  I pick a witches-brewed version of yards per target. It works for me, but it may not work for you.

Depth of Target

Once you’ve “solved” your dependent variable problem, you run headlong into another one.  Generally, whatever DV you choose is somehow correlated with depth of target.  Wide receivers that get thrown to farther down the field rack up more yards, generally get more touchdowns (that one is a bit tenuous, but I digress), have more yards per reception, and more yards per target.  So now you’re stuck with a problem of understanding how the wide receiver fits into the offensive system regarding average depth per target.  Does this receiver have low yards per target because they are not a particularly good receiver or because they are consistently being asked to be a “chain-mover” out of the slot?  This calculation is impossible in some circumstances and tricky even with witches-brewed data.

Small Effect Size

You know what actually predicts production at receiver? Targets. End. This is a graphic I made showing the relationship between targets and yards in college football.

Targets accounts for around 75-80% of the variance in yards, which means that there isn’t much variance left for differences in ability to do any work.  You could have a pretty decent receiver buried on a roster and they won’t look like much at all *cough cough Jarius Wright cough cough*  And the reverse is also true.  A relatively poor receiver could get a lot of targets and look like a golden god.

So, I wanted to post about wide receivers.  I ended up getting frustrated at the position and writing about my frustration.

What Exactly Do We Know Part III: The Nightmare Edition

Categories: Decision Making, NFL
Tags: No Tags
Comments: No Comments
Published on: October 28, 2014

This will be the final installment in my “What Exactly Do We Know” series. I think I’ve beat this horse enough that it’s about to fall over. But I need to talk about one final aspect of statistical reasoning and knowledge derived from data analysis. And this one is the creeping horror that should keep us all up at night. At the very least, this would keep me up at night if I were advising an actual NFL team. I’m going to being explaining the horror by having you imagine a job interview.

Actually, I’m going to ask you to imagine the lack of a job interview. How would you feel if you knew you were a top three candidate for a job, but the company called you one day saying “We don’t do interviews. We’ve already made our selection and we selected someone else because they had a higher college GPA.” When I ask my students to imagine this scenario, they say they’d be annoyed. They talk about how a particular score on a test doesn’t define them and if only they could get an interview they could prove their abilities and their worth.

However, if you’re trying to assess abilities and skills, evaluating on college GPA is actually the best way to get the skills and abilities you’re interested in. In fact, trying to assess abilities and skills with an unstructured conversation is one of the best ways to introduce unintended and significant bias into your decision making process. Most large, modern organizations don’t even use a conversation-style interview to assess skills anymore. Conversation-style interviews are done to only answer whether the person interviewing you could stand working with you for a day. But I digress.

I bring up job interviews because they are a fascinating point of the employee selection process. If used in the old let’s-chat-for-20-minutes way, the interviewer is unlikely to see the person’s worth with any form of accuracy. Which brings us back to football.

Football is an amazingly interesting game because of how interdependent all the action is. However, the interdependence leaves us with a fundamental problem. Can someone looking from outside the situation truly see what is actually happening in a quarterback-wide receiver connection? I looked in the published academic literature and couldn’t find the study that directly answers that question. I’m running the study in my lab right now, but I won’t have an answer for you for a long while. I haven’t looked at the data yet, but the tangentially related studies all seem to indicate that the answer is “No, we can’t see who is responsible for what when looking from the outside.” And, assuming I’m right, how then can we trust the opinion of any talent evaluator that doesn’t attempt to systematically control for such biases? Can even the most relevant talent evaluators, namely those that make personnel decisions for NFL teams, be trusted to make the right evaluation?

My top quarterback prospects from the 2014 draft were Nathan Scheelhaase and Keith Price. At the moment, neither of these players is on an NFL roster. The internet currently does not record what Scheelhaase is up to, but Prince is a quarterback – a backup for the Saskatchewan Roughriders in the Canadian Football League. So…what are we to make of this?

Keith Price 2013.jpg

Let’s say I’m right and we can’t trust talent evaluators that don’t use data to control the biases. This means that I can put out a list of prospect, those prospects can go out into the league and get evaluated. In the case of Price, he was evaluated by two of the best in the business – the Seahawks and Patriots.   Neither team desired his services which is how he ended up in Canada. But according to the theory we put out in the first paragraph, the fact that he didn’t get picked up doesn’t mean anything. We already believe that the evaluators can’t control a human bias. Hopefully you understand that this is a very advantageous rhetorical position to be in. How can you convince me that I’m wrong? What evidence would I accept if I won’t accept the pre-season evaluation of two teams who I have already stated I believe a two of the best in the league in evaluating talent?

The only evidence that the model will currently accept is on-field, regular season outcomes. And not only that, but I would need a lot of attempts to actually consider the notion I’m wrong. Which is why I would lose sleep if I worked for an NFL team. I’d have to trust the wings of Bayesian statistics to a degree I’ve never had to in the past. How terrifying would that be?

What Exactly Do You Know, Part II: The “Is He Your Cousin or Something?” Edition

Comments: No Comments
Published on: October 21, 2014

In last week’s post, I discussed the concept of what statistical inference actually tells you and how it’s boring and cumbersome to talk about it accurately, so analysts often shorten the conversation so they can actually talk with real people about something interesting. Today we take a slightly different tack regarding what exactly we know. Our example for this week is Minnesota Vikings quarterback Christian Ponder.

Christian Ponder close-up.jpg

If you’ve been reading for a while, you know that I was actually a fan of Ponder for a long time. Or, at the very least, I didn’t hate him with every fiber of my being like every other Vikings fan seemed to. I called him “not the problem in Minnesota” instead pointing to the largely ineffective receiving corps. I was talking to my neighbor before the season started. I said that Ponder is not the problem. We had this long and somewhat loud conversation about how I have to be wrong about him because everyone was giving up on Christian Ponder. Even Paul Allen – the radio play-by-play announcer for the Vikings – a guy who has never in his life given up on anyone in a purple jersey had given up on Christian Ponder. When I persisted that Ponder wasn’t the problem, my neighbor ended the conversation by saying, “You’re the only guy I know saying nice things about Ponder. Is he your cousin or something?” At the time the comment made me laugh. Then the Thursday night game against the Packers happened. I had to think more about this and examine what I know and what I don’t know about Christian Ponder in particular and the game of football in general.

So why was I so adamant that Ponder wasn’t the problem? Because, for all his faults, Ponder has one singular but important ability. He is rather accurate for an NFL quarterback. He’s not super-star Peyton Manning accurate, but he can get a football into a receiver’s hands slightly better than the average NFL quarterback. And why do I care so much about accuracy and nothing else? Because it’s the only quarterback ability I’ve found at the NFL level that will predict useful outcomes. Nothing else comes back predictive. Not a quantification of arm-strength, not Wonderlich scores, nothing at the combine, nothing but accuracy predicts NFL level outcomes.

And now we have another trap that analysts can fall into, a trap that is particularly present and meaningful for the NFL. I can’t find a predictive effect of my in-house metric that I think measures arm strength (let’s ignore the measurement point of “how do we know this thing is really arm strength” for now. It’s important but not where we’re going here). So I don’t find this effect. There are a couple possibilities why. The first possibility is the one that brings the page views and the loud conversations – that Arm Strength isn’t an important thing. However, another interpretation is that the lack of data at the NFL level makes finding the effect of arm strength insanely difficult.

Think about it like this. Imagine I told you that there was gold to be found in the body of water closest to you. To me that body of water is a river, so for the rest of this example I’ll be talking about a river. But maybe for you it’s a lake or an ocean or your friend’s bathtub. Whatever. You want to find this gold because you think having gold would be better than not having gold. So you go out and buy all the equipment necessary to pan for gold. You get the sorter pieces and the dirt sucker and everything else and you go stand in the river for a few hours and try to find this gold. Now, if you stood in the same spot panning for gold for four hours and didn’t find gold, would it be reasonable for anyone to assume that I’m wrong and that there is no gold in the river?

 

Crude Drawing of Where Gold is in a Fictional River
Crude Drawing of Where Gold is in a Fictional River

No, it would be ridiculous to say that. Maybe you were panning in the wrong spot. Maybe the screen you were using was too big and all the gold was little and slipping through. There could be many reasons why you didn’t find gold in the river.

Analytical findings are like gold. Just because you don’t find one, doesn’t mean that they aren’t there. This is a concept called “statistical power” and in the NFL it’s a huge problem. Our ability to find effects generally increases the more data we have. Think of it like this – more data makes our gold panning screens smaller. It allows us to find ever smaller nuggets of gold. In the NFL, the data is very sparse. There are only 32 teams playing 16 games each with maybe 30 passing attempts in each game. This pales in comparison to basketball’s 82 games and baseball’s 162. Compared to other sports, an effect in the NFL has to be fairly large before our screens will catch it. There is so little data coming from the NFL that it’s possible an arm-strength effect exists but there just isn’t enough data to find it.

So, after the Thursday night Ponder debacle, I went on a quest for more power. And in football, if you want more statistical power you need to look at the college level. With many many more teams we suddenly have a lot more power in our data set. I spent most of my summer calculating the same arm-strength metric for every NCAA FBS level quarterback and I ran the same model to see if arm-strength, along with accuracy, can predict useful quarterback outcomes. Low and behold, it does (said the amazed analyst and no one else). Ponder fairs very well on accuracy, but he suffers horribly on arm-strength. With this lesson learned, it’s time to quit dying trying to take the Ponder hill. Ponder is a problem for the Vikings offense. One of many, many problems.

Bills Bench E.J. Manuel: What’s the Plan?

Comments: No Comments
Published on: September 30, 2014

The Buffalo Bills have benched their starting quarterback and hope of the franchise, 2013 edition, E.J. Manuel. We don’t know if we’ll ever see him on a professional football field again. As of this writing, Manuel has a career passer rating of 78.5 and a career adjusted net yards per attempt (ANY/A) of 5.9. But this post isn’t about E.J. Manuel and what he has or hasn’t done on the football field.

My prediction of where E.J. Manuel would be after four years in the league was a passer rating of 74.8 with an ANY/A around 4.8. By my estimation he’s performed right about where I expected, maybe even a little better on ANY/A. But one case doesn’t prove the worth of a process and this post isn’t about me.

This post is about the Bills’ front office. I’m very curious what they expected of E.J. Manuel. In fact, I’m curious what they expected of their entire football team. Did they believe that, in 2012 when they went 6-10 that they were just a few players away from contending for a playoff spot? Did they truly believe that when they haven’t had a winner record since 2004?

I wonder what they expected on draft day in 2013. They drafted a new quarterback, certainly, but they’d also drafted a couple wide receivers in Robert Woods and Marquise Goodwin. I wonder if they considered how difficult it would be to evaluate the performance of a new quarterback when both quarterback and receivers were changing. I wonder how much hope existed in the draft room on that day and on what evidence that hope rested. I wonder if they felt the same way at the end of the 2014 draft when they had given away their 2015 first round pick to Cleveland to draft another new wide receiver in Sammy Watkins.

I wonder if, when making the decision to bench E.J. Manuel, the decision makers in the Bills organization thought about benching Robert Woods instead, a player I have currently rated as the 203rd best pass receiver in the NFL. I wonder if someone is thinking, “We threw the ball Wood’s way 12 times this week and gained a total of 17 yards on those 12 attempts. I wonder if the receiver had any impact at all on such a stat line.”

I wonder what the decision makers in Buffalo expect from Kyle Orton, a player whose career per attempt statistics look remarkably similar to the guy he’s replacing. What exactly do they think is going to happen here? We have a pretty good idea what Kyle Orton is going to do with a football in his hands. And we also know that Orton is old enough and experienced enough that he probably isn’t going to get much better. Manuel, whatever his faults might have, at least has a chance to improve.

So much in the sports world comes down to expectations. We expect E.J. Manuel to be good because he was the first quarterback taken in the 2013 draft, and the only quarterback taken in the first round. But why did we expect so much of E.J. Manuel? Because of how much someone else valued him? Because of what we thought we saw during his college days? Sadly, human minds cannot correctly weigh those expectations. Data is needed so that we don’t let our expectations interfere with our ability to make the best decision we can at this exact moment.

Mostly, Buffalo Bills decision makers, I wonder…

 

page 1 of 1
Welcome , today is Saturday, June 24, 2017