Never judge a goal keeper by his saves

Sometimes analysis and football intuition fit nicely together, and in those situations writing analysis pieces is easy. Sometimes they don’t, and writing gets tougher.

I’ve been thinking for most of the past weeks on goal keeper analysis. A topic that seems as simple as it gets, but as we’ll find out in this post, is actually a difficult one to get your head around and do properly.

GK saveAccording to the all-knowing Wikipedia, a goal keeper is “a designated player charged with directly preventing the opposing team from scoring by intercepting shots at goal”.

So, what could be more difficult than assessing how many of those shots end up as goals and, voilà, here’s our goal keeper analysis?

Let me start with a poll question. No need to fill out the answer, just take a little bit of time to make up which answer you think is correct.

The best way to identify goal keeping talent is…

  1. Percentage of shots saved
  2. Percentage of shots on target saved
  3. Percentage of shots saved with a correction for shot quality
  4. Other

 

In my personal history in football analysis I’ve gone from A to B, back to A, to C.

At C I’ve spent most of this season, but some background work I’ve done these past weeks have moved me further down, to D.

Yes, in my view, goal keeper analysis cannot reasonably be done on the basis of analyzing saves.

Now, that statement requires a bit of back up, so here we go. In the remaining part of this article we’ll analyze goal keepers in the top-5 leagues (England, Spain, Italy, Germany and France), who have faced at least 100 shots in two consecutive seasons (2012-13 and 2013-14), with the same club. To my idea, this is the best sample to use, to prevent keepers switching teams from screwing up the sample, and to prevent keepers with low numbers from doing the same.


Percentage of shots saved

We’ll start with raw saves percentage. This is the easiest parameter to collect, and probably the most used tool to evaluate goal keepers. It also ties in nicely with our intuition that good goal keepers stop a higher proportion of shots than bad goal keepers.

GK save percentage 03 februari 2014

The horizontal axis shows save percentage in the first year, and the vertical axis shows save percentage in the subsequent year. Remember, these are all goal keepers playing two seasons for the same club.

The connection is not very strong, but it’s not totally absent either. Generally, goal keepers who noted good saves percentages in the first year, noted better saves percentages in the second year, but the spread is huge. This makes it unreliable to estimate the second year’s saves percentages on the basis of the first year’s saves percentages. The repeatability of goal keepers saves percentage is poor. In general, if your stat has a poor repeatability, it’s useful to describe what has happened, but very misleading to assume that things will happen along the same lines in the future.

These numbers correspond with the excellent and far under viewed work by James Grayson, who found a similarly poor relation in a much larger set, matching teams in one season and the next.

 

Percentage of shots on target saved

Let’s move a little step forward and isolate shots on target. Some people advocate to use this over raw saves percentage, since goal keepers are hardly responsible for off target shots. In theory, though, keepers may take responsibility for some off target shots. By approaching a striker they could disrupt shot placement, or by reputation alone they could force strikers to try and find more difficult corners of the goal. Just raising a few hypotheses here.

GK save percentage SoT 03 februari 2014Again, first year performance is plotted on the horizontal axis, with second year performance on the vertical axis. The connection is even weaker for saves percentage of on target shots than it is for saves percentage of all shots conceded. Let’s save the debate until after the next plot.

 

Percentage of shots saved with a correction for shot quality

The third analysis uses shot quality. Based on our Expected Goals (ExpG) model, each shot is assigned a chance of ending up in goal, based on shot location, shot type and several other factors. This helps to control for the difficulty goal keepers have to make the save. In theory, this analysis is the best test for shot stopping quality, since it removes the fact that some keepers face tougher shots than others.

Goals conceded above replacement identifies how many goals a keeper conceded above or below the value of Expected Goals per 100 shots faced.

GK CAR 03 februari 2014After correcting for shot quality, all connection between first year performance and second year performance is lost. A goal keeper who over performed in the first year, has an equal chance of over performing in the second year as a goal keeper who under performed in the first year.

The most intriguing part of this rather shocking conclusion is that this knowledge is already out there, yet people continue to analyze goal keepers on the basis of saves. Again, I’m pointing you towards James Grayson, who, with smaller numbers taken from a Paul Riley post, found no correlation between goal keeper saves percentages in one season and the next after correction for shot location.

 

Shot quality

Please allow me to add one more scatter plot. This time, I’ve linked saves percentage and ExpG per shot, to show the strong link between those two.

GK save percentage and ExpG 03 februari 2014No goal keepers that faced shots higher than 0.11 ExpG noted a saves percentage over 92%, and no goal keepers that faced shots lower than 0.09 ExpG noted a saves percentage below 90%.

 

In the end

Putting all four plots together, this is compelling evidence to ignore each and every analysis using goal keeper saves percentage. The only, weak, link between goal keeper saves percentages (first graph) is driven by the quality of shots allowed. Some teams tend to face higher quality shots than others, therefore some goal keepers tend to have higher saves percentages than others. Nothing more, nothing less. On top of that, there’s going to be a huge amount of variance in performance.

This does not mean that shot stopping is not a skill. It most definitely is. It just indicates that among all factors that dictate a goal keeper’s saves percentage, the spread of skill level in shot stopping among top level goal keepers is very close. Other factors that influence goal keeper saves percentage completely overshadow the effect of skill, most notably shot quality, as indicated by ExpG.

 

Goal keeping talent

GK save 2So, how to scout for goal keeping talent? Start by ignoring saves percentage and you’ll leave most of the scouting world behind. Scouts will be aiming at goal keepers who’ve had random high saves percentages in some season, but those goal keepers stand an equal chance next season compared to all other goal keepers. Goal keepers who’ve had the bad luck of noting a low saves percentage season will probably be undervalued by the market.

What signs to look for, if not saves percentage? This piece shows compelling evidence against saves percentages, but it does not say that all goal keepers are equal. Far from that. It may well be that better goal keepers face less shots or shots with a lower ExpG. Better goal keepers will give up less rebound chances, less corners, claim more crosses, distribute balls better or sweep up nicely behind the defense.

All this stuff can be counted, but it’ll be hard to separate it from the effort of defenders. We’ll get to that in time. In the meantime, don’t let yourself be fooled by saves percentages.

19 thoughts on “Never judge a goal keeper by his saves

  1. Lars

    The third analysis/chart is the most interesting in my eyes, ExpG conceded minus goals conceded, that is it. Why? Because it indirectly includes so much more than the rest.

    Saved shots including shot quality, definitely, but also shots wide for example can be a metric for a good goalkeeper if strikers aim too well because the goalkeeper is well positioned. I am sure there is more.

    I wonder why the correlation between seasons is practically nonexistant.

    I can think of three reasons.

    1. Most goalkeepers are approximately equally good
    2. Goalkeepers’ form fluctuates a lot
    3. Sample too small

    2 and 3 however would explain that the correlation is small but this one here is tiny! Surprising result.

    Reply
  2. Joe

    Nice work. As a fan of baseball this reminds me of the dilemma that faces the statistical analysis of fielding defense.

    Fielding percentage, which is similar to save percentage, doesn’t really account for how hard a ball is hit, where it was hit, and in what situation. Even then the fielding percentage is a much bigger sample than save percentages.

    Most ballparks have now installed cameras and other devices to measure the vector of the ball to try and come up with a baseline of when a ball should be successfully fielded by an “average” player.

    Not a perfect comparison but there are tons of similarities.

    Reply
  3. staty

    I just can’t believe that a keeper’s saving values are so inconsistent from one season to the following one. There must be any metric that is a good measurement for a goalkeeper’s quality.

    I think that a version of “ExpG conceded-goals conceded” could be a solution: But not only adapted to shooting position and sort of assist, but also to the position where the shot hits the target. Perhaps, the shot’s speed has also to be taken into account.
    Intuitively, I’d say that a matrix consisting of those three factors (at least the first two have to be nitroduced) would be better than the existing stuff. Would be interesting to read some research about these issues.

    Reply
  4. Pingback: StatsBomb | Liverpool: An Analytical look at 2013/14 – A Missed Opportunity?

  5. Pingback: My Football Ramble: Using stats | Myfootballramble

  6. Ben

    While judging keepers might be very hard, at least you have a very good focal point: Manuel Neuer. Until your model does not cover every facet of his play, it is not complete. Also, if he doesn’t win, it’s faulty. I challenge you to prove the opposite (and to be honest, I’d lose gladly just to read the articles).

    Reply
  7. Pingback: ‘NOT JUST A SHOT-STOPPER’: CLAIMS% AS A MEASURE OF GOALKEEPER AERIAL PROWESS | outsiders | in

  8. Pingback: Introducing Goalkeeper radars and GK-score – Radar Love, flaws and conundrums | Il Trovaspazi

  9. Pingback: Jaden Smith vs Hugo Lloris: An Analysis of Punch%. | outsiders | in

  10. Pingback: Introducing RPI: Towards a repeatable metric of goalkeeper passing | outsiders | in

  11. Pingback: Why Leicester Might be Wise to Cash in on Kasper Schmeichel | outsiders | in

  12. Pingback: Data-analyse als toekomst van ‘ons’ voetbal – HowStatsWinFootball

  13. Pingback: Data-analyse als toekomst van ‘ons’ voetbal – HowStatsWinFootball

  14. Pingback: The Worst XI of the Rückrunde - Part 1 - Goalkeeper |

  15. Pingback: The Worst XI of the Bundesliga Rückrunde – Part 1 – Goalkeepers

  16. Pingback: Someone sign Mannone! A budget method of ranking shot-stoppers | outsiders | in

Leave a Reply