This will be the final installment in my “What Exactly Do We Know” series. I think I’ve beat this horse enough that it’s about to fall over. But I need to talk about one final aspect of statistical reasoning and knowledge derived from data analysis. And this one is the creeping horror that should keep us all up at night. At the very least, this would keep me up at night if I were advising an actual NFL team. I’m going to being explaining the horror by having you imagine a job interview.
Actually, I’m going to ask you to imagine the lack of a job interview. How would you feel if you knew you were a top three candidate for a job, but the company called you one day saying “We don’t do interviews. We’ve already made our selection and we selected someone else because they had a higher college GPA.” When I ask my students to imagine this scenario, they say they’d be annoyed. They talk about how a particular score on a test doesn’t define them and if only they could get an interview they could prove their abilities and their worth.
However, if you’re trying to assess abilities and skills, evaluating on college GPA is actually the best way to get the skills and abilities you’re interested in. In fact, trying to assess abilities and skills with an unstructured conversation is one of the best ways to introduce unintended and significant bias into your decision making process. Most large, modern organizations don’t even use a conversation-style interview to assess skills anymore. Conversation-style interviews are done to only answer whether the person interviewing you could stand working with you for a day. But I digress.
I bring up job interviews because they are a fascinating point of the employee selection process. If used in the old let’s-chat-for-20-minutes way, the interviewer is unlikely to see the person’s worth with any form of accuracy. Which brings us back to football.
Football is an amazingly interesting game because of how interdependent all the action is. However, the interdependence leaves us with a fundamental problem. Can someone looking from outside the situation truly see what is actually happening in a quarterback-wide receiver connection? I looked in the published academic literature and couldn’t find the study that directly answers that question. I’m running the study in my lab right now, but I won’t have an answer for you for a long while. I haven’t looked at the data yet, but the tangentially related studies all seem to indicate that the answer is “No, we can’t see who is responsible for what when looking from the outside.” And, assuming I’m right, how then can we trust the opinion of any talent evaluator that doesn’t attempt to systematically control for such biases? Can even the most relevant talent evaluators, namely those that make personnel decisions for NFL teams, be trusted to make the right evaluation?
My top quarterback prospects from the 2014 draft were Nathan Scheelhaase and Keith Price. At the moment, neither of these players is on an NFL roster. The internet currently does not record what Scheelhaase is up to, but Prince is a quarterback – a backup for the Saskatchewan Roughriders in the Canadian Football League. So…what are we to make of this?
Let’s say I’m right and we can’t trust talent evaluators that don’t use data to control the biases. This means that I can put out a list of prospect, those prospects can go out into the league and get evaluated. In the case of Price, he was evaluated by two of the best in the business – the Seahawks and Patriots. Neither team desired his services which is how he ended up in Canada. But according to the theory we put out in the first paragraph, the fact that he didn’t get picked up doesn’t mean anything. We already believe that the evaluators can’t control a human bias. Hopefully you understand that this is a very advantageous rhetorical position to be in. How can you convince me that I’m wrong? What evidence would I accept if I won’t accept the pre-season evaluation of two teams who I have already stated I believe a two of the best in the league in evaluating talent?
The only evidence that the model will currently accept is on-field, regular season outcomes. And not only that, but I would need a lot of attempts to actually consider the notion I’m wrong. Which is why I would lose sleep if I worked for an NFL team. I’d have to trust the wings of Bayesian statistics to a degree I’ve never had to in the past. How terrifying would that be?