Sometimes analysis and football intuition fit nicely together, and in those situations writing analysis pieces is easy. Sometimes they don’t, and writing gets tougher.
I’ve been thinking for most of the past weeks on goal keeper analysis. A topic that seems as simple as it gets, but as we’ll find out in this post, is actually a difficult one to get your head around and do properly.
So, what could be more difficult than assessing how many of those shots end up as goals and, voilà, here’s our goal keeper analysis?
Let me start with a poll question. No need to fill out the answer, just take a little bit of time to make up which answer you think is correct.
The best way to identify goal keeping talent is…
- Percentage of shots saved
- Percentage of shots on target saved
- Percentage of shots saved with a correction for shot quality
In my personal history in football analysis I’ve gone from A to B, back to A, to C.
At C I’ve spent most of this season, but some background work I’ve done these past weeks have moved me further down, to D.
Yes, in my view, goal keeper analysis cannot reasonably be done on the basis of analyzing saves.
Now, that statement requires a bit of back up, so here we go. In the remaining part of this article we’ll analyze goal keepers in the top-5 leagues (England, Spain, Italy, Germany and France), who have faced at least 100 shots in two consecutive seasons (2012-13 and 2013-14), with the same club. To my idea, this is the best sample to use, to prevent keepers switching teams from screwing up the sample, and to prevent keepers with low numbers from doing the same.
Percentage of shots saved
We’ll start with raw saves percentage. This is the easiest parameter to collect, and probably the most used tool to evaluate goal keepers. It also ties in nicely with our intuition that good goal keepers stop a higher proportion of shots than bad goal keepers.
The horizontal axis shows save percentage in the first year, and the vertical axis shows save percentage in the subsequent year. Remember, these are all goal keepers playing two seasons for the same club.
The connection is not very strong, but it’s not totally absent either. Generally, goal keepers who noted good saves percentages in the first year, noted better saves percentages in the second year, but the spread is huge. This makes it unreliable to estimate the second year’s saves percentages on the basis of the first year’s saves percentages. The repeatability of goal keepers saves percentage is poor. In general, if your stat has a poor repeatability, it’s useful to describe what has happened, but very misleading to assume that things will happen along the same lines in the future.
These numbers correspond with the excellent and far under viewed work by James Grayson, who found a similarly poor relation in a much larger set, matching teams in one season and the next.
Percentage of shots on target saved
Let’s move a little step forward and isolate shots on target. Some people advocate to use this over raw saves percentage, since goal keepers are hardly responsible for off target shots. In theory, though, keepers may take responsibility for some off target shots. By approaching a striker they could disrupt shot placement, or by reputation alone they could force strikers to try and find more difficult corners of the goal. Just raising a few hypotheses here.
Again, first year performance is plotted on the horizontal axis, with second year performance on the vertical axis. The connection is even weaker for saves percentage of on target shots than it is for saves percentage of all shots conceded. Let’s save the debate until after the next plot.
Percentage of shots saved with a correction for shot quality
The third analysis uses shot quality. Based on our Expected Goals (ExpG) model, each shot is assigned a chance of ending up in goal, based on shot location, shot type and several other factors. This helps to control for the difficulty goal keepers have to make the save. In theory, this analysis is the best test for shot stopping quality, since it removes the fact that some keepers face tougher shots than others.
Goals conceded above replacement identifies how many goals a keeper conceded above or below the value of Expected Goals per 100 shots faced.
After correcting for shot quality, all connection between first year performance and second year performance is lost. A goal keeper who over performed in the first year, has an equal chance of over performing in the second year as a goal keeper who under performed in the first year.
The most intriguing part of this rather shocking conclusion is that this knowledge is already out there, yet people continue to analyze goal keepers on the basis of saves. Again, I’m pointing you towards James Grayson, who, with smaller numbers taken from a Paul Riley post, found no correlation between goal keeper saves percentages in one season and the next after correction for shot location.
Please allow me to add one more scatter plot. This time, I’ve linked saves percentage and ExpG per shot, to show the strong link between those two.
In the end
Putting all four plots together, this is compelling evidence to ignore each and every analysis using goal keeper saves percentage. The only, weak, link between goal keeper saves percentages (first graph) is driven by the quality of shots allowed. Some teams tend to face higher quality shots than others, therefore some goal keepers tend to have higher saves percentages than others. Nothing more, nothing less. On top of that, there’s going to be a huge amount of variance in performance.
This does not mean that shot stopping is not a skill. It most definitely is. It just indicates that among all factors that dictate a goal keeper’s saves percentage, the spread of skill level in shot stopping among top level goal keepers is very close. Other factors that influence goal keeper saves percentage completely overshadow the effect of skill, most notably shot quality, as indicated by ExpG.
Goal keeping talent
So, how to scout for goal keeping talent? Start by ignoring saves percentage and you’ll leave most of the scouting world behind. Scouts will be aiming at goal keepers who’ve had random high saves percentages in some season, but those goal keepers stand an equal chance next season compared to all other goal keepers. Goal keepers who’ve had the bad luck of noting a low saves percentage season will probably be undervalued by the market.
What signs to look for, if not saves percentage? This piece shows compelling evidence against saves percentages, but it does not say that all goal keepers are equal. Far from that. It may well be that better goal keepers face less shots or shots with a lower ExpG. Better goal keepers will give up less rebound chances, less corners, claim more crosses, distribute balls better or sweep up nicely behind the defense.
All this stuff can be counted, but it’ll be hard to separate it from the effort of defenders. We’ll get to that in time. In the meantime, don’t let yourself be fooled by saves percentages.