What Exactly Do You Know, Part II: The “Is He Your Cousin or Something?” Edition

Comments: No Comments
Published on: October 21, 2014

In last week’s post, I discussed the concept of what statistical inference actually tells you and how it’s boring and cumbersome to talk about it accurately, so analysts often shorten the conversation so they can actually talk with real people about something interesting. Today we take a slightly different tack regarding what exactly we know. Our example for this week is Minnesota Vikings quarterback Christian Ponder.

Christian Ponder close-up.jpg

If you’ve been reading for a while, you know that I was actually a fan of Ponder for a long time. Or, at the very least, I didn’t hate him with every fiber of my being like every other Vikings fan seemed to. I called him “not the problem in Minnesota” instead pointing to the largely ineffective receiving corps. I was talking to my neighbor before the season started. I said that Ponder is not the problem. We had this long and somewhat loud conversation about how I have to be wrong about him because everyone was giving up on Christian Ponder. Even Paul Allen – the radio play-by-play announcer for the Vikings – a guy who has never in his life given up on anyone in a purple jersey had given up on Christian Ponder. When I persisted that Ponder wasn’t the problem, my neighbor ended the conversation by saying, “You’re the only guy I know saying nice things about Ponder. Is he your cousin or something?” At the time the comment made me laugh. Then the Thursday night game against the Packers happened. I had to think more about this and examine what I know and what I don’t know about Christian Ponder in particular and the game of football in general.

So why was I so adamant that Ponder wasn’t the problem? Because, for all his faults, Ponder has one singular but important ability. He is rather accurate for an NFL quarterback. He’s not super-star Peyton Manning accurate, but he can get a football into a receiver’s hands slightly better than the average NFL quarterback. And why do I care so much about accuracy and nothing else? Because it’s the only quarterback ability I’ve found at the NFL level that will predict useful outcomes. Nothing else comes back predictive. Not a quantification of arm-strength, not Wonderlich scores, nothing at the combine, nothing but accuracy predicts NFL level outcomes.

And now we have another trap that analysts can fall into, a trap that is particularly present and meaningful for the NFL. I can’t find a predictive effect of my in-house metric that I think measures arm strength (let’s ignore the measurement point of “how do we know this thing is really arm strength” for now. It’s important but not where we’re going here). So I don’t find this effect. There are a couple possibilities why. The first possibility is the one that brings the page views and the loud conversations – that Arm Strength isn’t an important thing. However, another interpretation is that the lack of data at the NFL level makes finding the effect of arm strength insanely difficult.

Think about it like this. Imagine I told you that there was gold to be found in the body of water closest to you. To me that body of water is a river, so for the rest of this example I’ll be talking about a river. But maybe for you it’s a lake or an ocean or your friend’s bathtub. Whatever. You want to find this gold because you think having gold would be better than not having gold. So you go out and buy all the equipment necessary to pan for gold. You get the sorter pieces and the dirt sucker and everything else and you go stand in the river for a few hours and try to find this gold. Now, if you stood in the same spot panning for gold for four hours and didn’t find gold, would it be reasonable for anyone to assume that I’m wrong and that there is no gold in the river?

 

Crude Drawing of Where Gold is in a Fictional River
Crude Drawing of Where Gold is in a Fictional River

No, it would be ridiculous to say that. Maybe you were panning in the wrong spot. Maybe the screen you were using was too big and all the gold was little and slipping through. There could be many reasons why you didn’t find gold in the river.

Analytical findings are like gold. Just because you don’t find one, doesn’t mean that they aren’t there. This is a concept called “statistical power” and in the NFL it’s a huge problem. Our ability to find effects generally increases the more data we have. Think of it like this – more data makes our gold panning screens smaller. It allows us to find ever smaller nuggets of gold. In the NFL, the data is very sparse. There are only 32 teams playing 16 games each with maybe 30 passing attempts in each game. This pales in comparison to basketball’s 82 games and baseball’s 162. Compared to other sports, an effect in the NFL has to be fairly large before our screens will catch it. There is so little data coming from the NFL that it’s possible an arm-strength effect exists but there just isn’t enough data to find it.

So, after the Thursday night Ponder debacle, I went on a quest for more power. And in football, if you want more statistical power you need to look at the college level. With many many more teams we suddenly have a lot more power in our data set. I spent most of my summer calculating the same arm-strength metric for every NCAA FBS level quarterback and I ran the same model to see if arm-strength, along with accuracy, can predict useful quarterback outcomes. Low and behold, it does (said the amazed analyst and no one else). Ponder fairs very well on accuracy, but he suffers horribly on arm-strength. With this lesson learned, it’s time to quit dying trying to take the Ponder hill. Ponder is a problem for the Vikings offense. One of many, many problems.

What Exactly Do You Know?

Categories: General Info, Statistics
Comments: 1 Comment
Published on: October 14, 2014

(This post was inspired, in part, by a post Matt Waldman posted on his website a few days ago. If you haven’t already, go check out his site. He does an amazing job over there. The post in question is titled “Deny Emotion and You Only See a Fraction of the Game.”)

The establishment in sports has, for a while now, been telling outsider-sports-analytics types that one of the main barriers to widespread acceptance of analytics rests on the ability of the quants to communicate with the non-quants in ways that the non-quants can understand. I’ve covered the problems of communicating information both to quants and to non-quants in the past. But Matt’s post about emotion made me realize something more.

His general point was about the role emotion plays in sports performance. Specifically, he was contemplating whether or not momentum exists in sports. He cites conversations with analytics experts on the subject of momentum. If you’re familiar with the argument, most analytics experts conclude momentum does not exist. This is a fairly standard finding across many sports – basketball’s hot hand being another classic example. Matt’s reaction to that conclusion is also fairly standard among non-analysts. His argument is that emotion, and the effects of emotion on situations in sports are obviously occurring. Anyone who denies that is missing a huge portion of the signal inherent in the game. At one point he suggests that analysts have possibly never put themselves in physically dangerous situations and felt the impact of emotion first hand.

So, two things with that summation, both having to do with communication. First, assuming experiences that someone has or hasn’t had is a raw nerve for me. The stereotype of the milquetoast academic who simulates experiences rather than having actual experiences looms large over me. I’m guessing it has something to do with my father telling me to put down the video games and go spend some time working on my grandfather’s farm. But, I know the comment wasn’t directed at me and it wasn’t malicious anyway, so let’s put aside any irritation that might shut our brains off. In fact, we need to keep our brains on if we’re going to truly examine the point Matt is trying to get at.

Analysts (myself included) are often guilty of a particular linguistic shorthand. Our job is to find predictive effects. Does changing the way a request is phrased reliably change donations to charity? If I know about your height, can I make an educated guess about your weight? That sort of thing. We can become so familiar, so practiced in that job that, when we talk to other people, we tend to shorten the description of what we’re talking about. When my neighbor asks me what I know about momentum in sports, I say “I’m trying to find out if momentum exists” and he gets excited and engages with the conversation. The problem for analysts, though, is that this gets the conversation off on a disingenuous foot. Because we’re not truly trying to find out if momentum exists. We’re trying to find out if the predictive effect of momentum on some other variable, like scoring, exists. But we repeat the phrasing about studying the existence of momentum so much that we can forget that other people see that collection of words as having a different meaning.

And when we analyze the results in sports, the results a pretty clear. The effect of momentum on any variable we look at is unpredictable. The same is true in the academic literature as well. We cannot predict the effect of emotion on motivation with any reliability. So, I agree that we should probably stop saying that momentum doesn’t exist. The subjective experience of the emotion of the game is a real thing that people feel.  But I will hold fast to the notion that predicting what will initiate a change in momentum and how a change in momentum will impact athletic performance is an unpredictable enterprise.

NCAA Quarterbacks: 2015 Draft Class

The quarterback situation for the 2015 draft class is looking very murky.  The 2012 draft class was a very unique class in that the highly talented players all managed to get drafted highly and to generate the needed playing time to demonstrate that talent.  I don’t see that happening with the 2015 draft class currently.  I see the 2015 class looking a lot more like the 2013 draft class.  Guys that can play exist in the pool, but who knows if they will get the playing time they need.

Revisiting the quarterbacks I’m watching section from earlier this year, I’m still high on Rakeem Cato.  I saw one feature story on him, but it had to do more with his background than with his ability as a passer.  Hopefully he gets some more attention for the latter.

One player we should keep a close eye on is Bo Wallace at Ole Miss.  I didn’t have him on my original list this year, but he’s having a very good season, both in terms of accuracy and in down the field throws.  That second part is important because he did not have a good season throwing down field last year.  Good to see him demonstrate that he actually has that ability.

Last but not least, someone to keep an eye on is Conner Halliday at Washington State.  Not saying I’m overly excited about him, but he does have something worth looking at.  I don’t know where he’ll end the season in my projections, but his current performance is at least elevating him from where he was.

Last but not least, looking far into the future, is anybody looking at this sophomore from Middle Tennessee?  His name is Austin Grammer.  A name you might want to get to know.

Bills Bench E.J. Manuel: What’s the Plan?

Comments: No Comments
Published on: September 30, 2014

The Buffalo Bills have benched their starting quarterback and hope of the franchise, 2013 edition, E.J. Manuel. We don’t know if we’ll ever see him on a professional football field again. As of this writing, Manuel has a career passer rating of 78.5 and a career adjusted net yards per attempt (ANY/A) of 5.9. But this post isn’t about E.J. Manuel and what he has or hasn’t done on the football field.

My prediction of where E.J. Manuel would be after four years in the league was a passer rating of 74.8 with an ANY/A around 4.8. By my estimation he’s performed right about where I expected, maybe even a little better on ANY/A. But one case doesn’t prove the worth of a process and this post isn’t about me.

This post is about the Bills’ front office. I’m very curious what they expected of E.J. Manuel. In fact, I’m curious what they expected of their entire football team. Did they believe that, in 2012 when they went 6-10 that they were just a few players away from contending for a playoff spot? Did they truly believe that when they haven’t had a winner record since 2004?

I wonder what they expected on draft day in 2013. They drafted a new quarterback, certainly, but they’d also drafted a couple wide receivers in Robert Woods and Marquise Goodwin. I wonder if they considered how difficult it would be to evaluate the performance of a new quarterback when both quarterback and receivers were changing. I wonder how much hope existed in the draft room on that day and on what evidence that hope rested. I wonder if they felt the same way at the end of the 2014 draft when they had given away their 2015 first round pick to Cleveland to draft another new wide receiver in Sammy Watkins.

I wonder if, when making the decision to bench E.J. Manuel, the decision makers in the Bills organization thought about benching Robert Woods instead, a player I have currently rated as the 203rd best pass receiver in the NFL. I wonder if someone is thinking, “We threw the ball Wood’s way 12 times this week and gained a total of 17 yards on those 12 attempts. I wonder if the receiver had any impact at all on such a stat line.”

I wonder what the decision makers in Buffalo expect from Kyle Orton, a player whose career per attempt statistics look remarkably similar to the guy he’s replacing. What exactly do they think is going to happen here? We have a pretty good idea what Kyle Orton is going to do with a football in his hands. And we also know that Orton is old enough and experienced enough that he probably isn’t going to get much better. Manuel, whatever his faults might have, at least has a chance to improve.

So much in the sports world comes down to expectations. We expect E.J. Manuel to be good because he was the first quarterback taken in the 2013 draft, and the only quarterback taken in the first round. But why did we expect so much of E.J. Manuel? Because of how much someone else valued him? Because of what we thought we saw during his college days? Sadly, human minds cannot correctly weigh those expectations. Data is needed so that we don’t let our expectations interfere with our ability to make the best decision we can at this exact moment.

Mostly, Buffalo Bills decision makers, I wonder…

 

Ryan Tannehill – Using Passer Rating to Evaluate Quarterbacks

Categories: NFL, Statistics
Comments: No Comments
Published on: September 24, 2014

Apparently Ryan Tannehill doesn’t have a high enough passer rating this season. At least, not high enough for his coach, Joe Philbin to commit to him being the starting quarterback for the Dolphins on two separate occations. I first saw this yesterday and assumed it was some thinly veiled motivational tool. But then it happened again today. Which doesn’t happen if you’re trying to use shame as a motivational tool…at least, not if you’re doing it right.

So let’s assume that Philbin is seriously considering benching Ryan Tannehill and putting in Matt Moore because Tannehill’s passer rating isn’t high enough. What are the implications of such a move?

First, Ryan Tannehill has not been good this year. In fact, he’s been one of the worst quarterbacks in the league over the first three games of the season, depending how you figure it. Philbin isn’t wrong that Tannehill’s passer rating is awful. However, and this is the really important part, who cares?

Whenever you start using numerical information to inform real world decisions, you must be incredibly careful to select the correct criterion variable. Criterion variables are the regression equivalent of dependent variables, a variable we care about but don’t mess with. We just allow it to vary naturally. Proper selection of your criterion variable is one of the most important decision you make when beginning a program of analysis. Every truth you uncover during your studies will only be true with respect to the criterion variable you chose. That single decision will color every single decision from the moment you make it on.

Returning to the last question I asked, who cares if Tannehill’s passer rating isn’t particularly good? Most people in the football analytics community would say no one should care. Pro Football Focus loves saying that passer rating tells you about the efficiency of the offense, but not the individual running it. Others like Chase Stuart at Football Perspective talk about how the weights assigned to passer rating are arbitrary and incorrect. These heavy hitters in the analytics community say that Philbin is using the wrong criterion variable to evaluate his quarterback. And I would largely agree with these perspectives.

So what does Philbin really want out of his quarterback? That’s the ultimate question in football. What criterion variable should we choose to actually understand the game and the people who play it? Those of us interested in understanding the game of football from an analytic perspective should be scrambling trying to understand everything we can about what different criterion variables tell us about the game. The correct selection of criterion variables is one of the most critical questions we face at this moment.  We know passer rating isn’t particularly good. ANY/A and ESPN’s QBR certainly make better cases as being good criterion variables, but the perfect variable doesn’t exist.

Ultimately, my guess is that Philbin would just like his offense to contribute to a win. But, if Joe Philbin is really going to use passer rating to evaluate quarterbacks, I could have told him to avoid Tannehill in the first place.

SeaWorld and the NFL

The orcas at SeaWorld are getting a new habitat. The new habitat will cost SeaWorld hundreds of millions of dollars and basically double the size of the habitat the orcas currently have.

Several events have conspired together to move SeaWorld toward building this habitat, but the catalyst can be traced back to a large male orca named Tilikum who murdered Dawn Brancheau, an experienced trainer who was working with him, in 2010.

A key point in the debate about whether or not “murdered” is an acceptable word for the events that transpired rests on whether or not any trainer at any level of experience is safe in the pool with an orca whale. In court, SeaWorld contended that trainers well versed in methods of animal learning and operant conditioning were perfectly capable of controlling a 3 ton whale. The Occupational Safety and Health Administration, on the other hand, contended that no amount of contact with orcas could be considered safe. After some legal wrangling, it was decided that any and all contact between orcas and trainers had to be done with a solid material, such as a concrete barrier, between the trainer and the whale. Direct contact was no longer allowed.

I’ve been returning to my feelings about the SeaWorld incident a lot during this week of terrible football fandom that just won’t end. I wonder about organized football, both in college and the NFL, from the perspective of a workplace. I think about the increased rates of brain damage, along with the recent instances of domestic violence, child abuse, and rape and I wonder what this game that I love to watch is doing to the individuals that get a chance to play it at a high level.

Last week I wrote about one of the great benefits to using statistical methods for employee selection. Basically, using math to find people means it is much easier to find other people should the ones you originally find not work out socially. That’s a great boon to employers in a typical workplace. They become less beholden to talent. Talented people no longer become insulated from the consequences of socially unacceptable acts. And, for any typical place of employment, that works quite well. However, I did not consider in that piece that the NFL is not a typical place of employment. Professional football has some troubling facts associated with it. As a psychology professor, I know all about what brain damage to the prefrontal cortex can do to an individual. Increased emotional reactivity and impulsivity are just the tip of the iceberg. I wonder if this game is destroying the lives of the people that play it. And I worry that anyone that works to identify and project success from one level of football to another, including myself, may be complicit in that destruction.

I am well aware that this blog isn’t particularly popular. As of this writing, I have 45 followers on Twitter and I get about 6 readers a day, 3 of which are spam crawlers. I am, at the moment, insulated from a looming ethical dilemma. No one with actual decision making power is calling me to learn my opinion on whether or not a particular player should be granted the “privilege” of continuing to play the game. The minute that happens, though, I will have an important decision to make. I will need to choose whether or not I should use my intellect to grant someone else the opportunity to potentially destroy theirs in the name of a sporting contest. I will need to decide if I believe the game of football can be played in a way that doesn’t destroy the lives of its players or if I believe my ability to identity talented football players is the same as placing the best trainers in a pool with 3 tons of socially maladjusted rage.

I do know one thing. Talking about football has never seemed more hollow than in these last 10 days. The heart-rune of my fandom is ripped. I’m not sure it will ever heal.

Beholden to Talented Shitheads: Why We Need Analytics

Categories: General Info, NFL
Comments: 1 Comment
Published on: September 9, 2014

I hope everyone is enjoying the new football season. I’m glad to see the Vikings are 1-0 and the defense looked good, although, it was against the Rams so I’m not sure that means all that much.

I don’t have much to talk about in the way of numbers today. We’ve got one week worth of NFL data which will tell us largely nothing about how the rest of the season will play out and we’ve got two weeks of college football data which will tell us something so minor that we probably shouldn’t bother right now.

Instead, I thought I would talk about one of the more important social issues surrounding football right now. I want to talk about Ray Rice and, specifically, what Ray Rice shows us about the importance of adopting analytic strategies for selecting members of organizations.

Many people think that businesses use analytic strategies like skill testing and personality testing because the tests tell you which individual is the most talented, most productive, most useful potential employee and the business then selects the person who comes out on top of the most important tests. And if you think that, you’d be sort-of right about how the process works, but you’d also be sort of wrong.

Most businesses that use analytic strategies use their tests not to find a single individual, but instead to narrow the pool of possible individuals. Tests are used to cull the group, but they generally aren’t used to make a final decision. High scores are necessary to land the job, but they aren’t sufficient. Once the tests identify the proper pool of applications comes the next, and most vital question an interviewing team can ask, “Can we all work with this person?” Fit within the work culture and ability to get along with co-workers is critical to building a functional organization. Any business using this strategy needs to be very careful that their answers to whether they can work with different people are not biased in ways that violate Civil Rights laws or any moral principles that the company holds to, but in general that’s how companies use tests to select employees. Test them all, generate a pool, but don’t select based solely on high scores but rather on more human elements.

That’s the first way analytics helps you build your organization. You can be sure of selecting talented people that are actually the kind of people you want to work with. And that could be important if you’re trying to build a football team. Many coaches seem to have very high minded policies about avoiding players with domestic violence histories. And while they seem to stick to those principles to greater or lesser degrees depending on the talent of the player in question, we can at least see how this would work. If your analytic strategy returns two players as equally likely to succeed and one of them has a history of domestic violence, you probably go with the other one. But that’s not why NFL teams need to quickly adopt analytics.

Using analytics to select employees is critical when one of your talented and valuable employees makes a mistake so horrendous, so unspeakable that it makes you rethink whether or not you would be able to work with that person ever again. Enter our connection to Ray Rice.

What Ray Rice did was unspeakable. But how the Ravens and the NFL responded to the situation is just as unspeakable. And while I can’t speculate on what was going through Rice’s head when he committed his act, I have been associated with enough employee selection meetings to have a guess at what the Ravens were thinking prior to cutting him.

The Ravens, and all NFL teams, are in an industry where talent is incredibly difficult to identify. Highly trained NFL scouts get evaluations of talent wrong every season. It’s a terrible job to try to be good at because almost no one truly knows what it takes to be a great football player. If the organization can’t reliably identify talent, it becomes very guarded about the talent that has fallen into its lap. And when organizations have limited confidence in their ability to find new talent, they are more willing to forgive egregious actions from the talent they actually have. In essence, organizations can become beholden to talented shitheads.

Selecting players using analytic strategies can break that cycle. When a talented member of the organization moves into territory that the rest of the organization can’t follow, it is a simple matter to separate from that person, regenerate a new pool of potential applicants, and begin the selection process all over again. We don’t have to run our rationalizer ragged trying to find reasons why Action X might be morally repugnant, but doesn’t justify removal of the person from the organization. Instead, the incentives for talented individuals to act like a shitheads evaporate. The organization can afford to be less risk-averse when problems with talented players emerge. If the Ravens had a large scale analytics-based selection process they could have cut Rice in February and found two or three shiny new running backs. Instead, we have the nonsense we all saw this week. Honestly, I fail to see how the status quo is better.

Final NFL Pre-Season Projections

Categories: Uncategorized
Tags: No Tags
Comments: No Comments
Published on: September 3, 2014

The NFL Season starts tomorrow.  As such, this will be my last revision of the preseason yardage projections.  To access the projections, either click on the link in the top banner or click on these links for

Quarterbacks

Wide Receivers

Tight Ends

A couple notes.

1)  Wes Welker is completely gone from these projections.  The news just came out that he’s out for 4 games due to some substance use, plus who knows how this concussion business will turn out.  Welker being gone actually helps Peyton Manning move into the top spot in yardage projections for quarterbacks.  Welker may be effective at getting you first downs, but he doesn’t do much for a quarterback’s YAC.  As such, we’re expecting Manning to get a few more yards with Welker out of the lineup than with him in.

2)  The Saints just resigned Robert Meachem (and surprisingly cut Ryan Griffin to do it).  Meachem has been a high quality receiver throughout his career, so his signing would be good news for Drew Brees.  He’s not on this list because there isn’t enough time to figure out where he finds a spot on the depth chart.  We will leave our conclusions about him to later in the season.

3) Very very very important to remember that the ability of this model to predict outsample date has not been validated.  We’re all going to be learning how this model does throughout the season.  Yay science.

 

NCAA Quarterbacks to Watch – 2014 Season

Categories: NCAA FBS, NFL Draft
Comments: 1 Comment
Published on: September 2, 2014

Week 1 of the college football season is in the books. I had intended to put together a post of the college quarterbacks I’m watching this year as NFL prospects. I did this last year and saw a lot of search interest in that poorly titled piece. Getting this piece out wasn’t near the top of the priority list, then the new semester started, I started working on a projection model for NFL production, and didn’t get around to putting out this piece until today. Sadly, this means I didn’t strike while the iron was hot in one specific case. I’ll take this as a lesson in timely publication.

Without further ado, the quarterbacks I’m watching for the 2014 NCAA season

Rakeem Cato, Marshall

Cato12.png

Cato is still my leader in the clubhouse for NFL quarterback prospects. I actually thought he declared for the draft last year. If he had, he would have been my #3 quarterback prospect then. Things are looking good for him and I’m excited to see what he’s got during his senior season.

Brent Hundley, UCLA

Brett Hundley.jpg

The first player on our list that many scouts will likely agree on. He’s currently second on my list of prospects, but had a really poor game by his standards last weekend. I’ll be watching this one to see which way it goes. Keep watching the season numbers page for more updates.

Shane Carden, East Carolina

Here’s someone you probably haven’t heard of. I’ve said it before, but one of the best things about using numbers to scout prospects is the ability to find gems that don’t get a large amount of T.V. time.  And the numbers say Carden definitely is worth a long look from professional scouts. I think you’ll be pleasantly surprised by what you see.

Sean Mannion, Oregon State

Sean Mannion.jpg

Mannion apparently really impressed some people over the summer at a Manning passing academy. I’m glad to hear it because I’ve been watching him with some interest since the beginning of last season. Hopefully he can continue the trend and keep trending up. I’ll be excited to see where he ends up.

Brandon Doughty, Western Kentucky

Yeah yeah, I know. I’m four days late and six touchdowns short on calling this one. Lesson learned, I suppose. I should probably also note that with last weekend’s performance Doughty launched himself from fifth to third on this list. Keep watching this guy.

So there you have it. My quick list of who I’m watching this season. As always, season numbers are available on the top ribbon.

Quick Update

Categories: Uncategorized
Tags: No Tags
Comments: No Comments
Published on: August 26, 2014

Nothing much new to report on the data front this week.  I updated the predictions for yardage totals recently.  The most important changes to the predictions are

1) Replace Sam Bradford with Shaun Hill as the Rams quarterback.  That change does not affect the predictions for the Rams receivers to a great degree.  My Twitter thoughts on the matter have not changed

What will change the predictions dramatically is getting some clarity as to who will get the lions share of the targets in St. Louis.  It seems as though the websites I access for information about depth charts have wildly different ideas about the Rams receiver depth chart.  This will probably update again before the season begins.

2)  Wes Welker completely removed.  I have no idea if Welker will come back or not.  The other day everything was “long-term health” and now today everything is “moving through the protocol.”  We’ll see, I guess.

«page 1 of 7
Welcome , today is Wednesday, October 22, 2014