Goalies aren't easy to evaluate. Or as some might say, they are voodoo. Introduced some time back by the war on ice crew(which will soon be defunct), and used by Nick Mercadante in his Mercad60, they split up shots into low, medium, and high danger based on the position of the shot and whether it was a rebound or a rush shot. And it's been used quite a bit. I've been thinking about it lately, and strangely, I've never actually seen anyone regress it. So I did so and I thought I might as well share it.
What we would expect, from intuition and what we know, is that: Low-Danger is almost all noise, Mid-Danger has a signal but it's weak, and High-Danger is the best and has a fair signal. Of course, we need to confirm this. My first to idea to test all this was to do it the conventional way. Take all goalies and line up all their shots. And run odd-even correlations until r=.5 (that being the correlation coefficient). The amount of shots needed to reach r=.5 would be how much we regress (and we would do so at league average). But the problem is that Sv% (in all zones) take time until a good signal is reached, and there are only so many goalies who have faced a lot of shots. We start to run out of goalies as the sample is too small (I was actually able to get a read on High-Danger Sv% and found it reached r=.5 at 1050......this makes sense as we'll see soon).
Thankfully, there are other ways. On way, which is easier and more convenient, is shown here by Tom Tango. I strongly recommend you read that before continuing (and I also recommend reading his blog in general). Ok (I'm assuming you read it), I ran the numbers on the last 4 years on goalies who faced at least 750 total shots. Here's what we find (It goes without saying....this is all Even Strength numbers). Also I only included sv% for the hell of it and for a self-check (I know it should be about 3000). The best way is regressing each zone independently.
n Sv% Low-Sv% Mid-Sv% High-Sv%
SD of z Scores: 76 1.307 1.136 1.033 1.298
# of shots regressed: 76 3534 3915 10127 1001
First thoughts from these numbers is that high danger and regular Sv% (it's near Tango's estimate) make sense. But low and mid seem to be in the wrong spots. Flip it and it makes sense, but it doesn't match up. So what I did was run it on the 4 years previous to that (2008-2012). And what we find is:
n Sv% Low-Sv% Mid-Sv% High-Sv%
SD of z Scores: 71 1.457 .942 1.268 1.264
# of shots regressed: 71 2638 -11565 1352 1400
For this one Sv% makes a little more sense and High-Danger stays about the same. But as you can easily tell Low-Danger is fucked up and Mid-Danger seems too low. So what we'll do is run them together. We'll just combine the data and see what we find (One note: The percentages used now is for the entire eight year period not each four individually).
n Sv% Low-Sv% Mid-Sv% High-Sv%
SD of z Scores: 147 1.4 1.046 1.16 1.295
# of shots regressed: 147 2838 12933 2165 1121
And these estimates make the most sense. Sv% is near the 3000 mark, Low-Danger is pretty much just noise, High-Danger is the best and has a fair signal. And Mid-Danger is a weaker signal.
Lastly, I'd like you to think of the spread for each danger zone sv%. Over the past three years the observed Standard Deviation for each one is: Low- .69%, Mid- 1.317%, High- 2.232%. And what we know is that the amount of noise decreases as we move to the right. So not only does the observed spread increase as we move to High-Danger but we can attribute more of that to talent (the standard deviation due to talent is higher). I think this is important in showing why we should be mainly focusing on High Danger sv% when it comes to evaluating goalies (obviously not only focus on it, just mostly) as it has the biggest spread in talent.
***All Data courtesy of War On Ice
For this who care.....the league average sv% for each zone (and in general) over the past 4 years:
Sv% Low-Sv% Mid-Sv% High Sv%
.923 .9738 .927 .8347
Tuesday, May 31, 2016
Friday, April 15, 2016
To Secondary Assist or not to Secondary Assist
Just over five years ago Eric Tulsky, who now works for the Carolina Hurricanes, published this article on assists. As you can see, he advocates dropping the secondary assist due to the amount of randomness in it (to be fair, I don't know his stance after publishing that). This is an older article, so I feel like I must as well update the numbers and the analysis and see what we find.
So I'm going to be using numbers from the past six years (2010-2016). I'm just going to do a simple year over year analysis like Eric did, but I'll split up between forwards and defensemen (I tried splitting up forwards between center and wingers but there was virtually no difference). Also the minimum amount players needed to have qualified for the analysis was 400 minutes for both years.
Repeatability
And you can see that in all categories, players who stayed on the same team showed better persistence. Let's look at forwards: For forwards, first assists goes slightly down. But you really see it in secondary assists, where it takes a bit of a tumble. And this of course results in assists as a whole grading out worse for forwards who change teams. Defensemen, on the other hand, really have their numbers lose a lot. Secondary assists are virtually random when changing teams. And surprisingly first assists take a big hit too. I really didn't expect that much of a difference.
One final thing though, let's look at what better predicts next years A60. First assists or total assists.
As you would imagine total assists edges out first assists in all cases except, oddly, defensemen who stay on the same team. I won't put much thought into that, it's nothing. Also, the edge in each case is really small. I would imagine that this being only one year worth of data contributes to this. As you would imagine, after getting a few years of playing time we can make a better estimate of secondary assists.
Conclusion
The numbers here are, overall, close to the one's shown by Eric T. five years back. And I think the best bet when looking at assists is focusing primarily on primary assists. But secondary assists still matter a little. One year of assists tells us more than primary assists. Not by much, but there is something. And given a few years, we'll get a better judge of a players secondary assist "talent". Just looking at the leaderboards for secondary assist over the past few years will tell you that it means something. Secondary assists may contain a lot of randomness, but they still matter. The gain over primary assists is minimal, but they shouldn't just be discarded as noise (just mostly noise).
**All data courtesy of Corsica.hockey
***Update:
This is a good article on secondary assist-http://fivethirtyeight.com/features/some-nhl-stars-get-more-assists-at-home-than-they-deserve/. The next step would probably be adjusting for rink bias. Also this
So I'm going to be using numbers from the past six years (2010-2016). I'm just going to do a simple year over year analysis like Eric did, but I'll split up between forwards and defensemen (I tried splitting up forwards between center and wingers but there was virtually no difference). Also the minimum amount players needed to have qualified for the analysis was 400 minutes for both years.
Repeatability
n
|
Position
|
A160
|
A260
|
A60
|
1402
|
Forwards
|
0.36
|
0.19
|
0.43
|
836
|
Defensemen
|
0.20
|
0.03
|
0.24
|
Here are the correlation coefficients for each metric. And I would say this all makes sense. First assists for both are more reliable than secondary assists. More so for forwards than for defensemen, for whom it's almost virtually random. And this is reflected in the total assists numbers. For defensemen it's only slightly higher than first assists. For forwards we see a a little bit more there.
Of course this isn't it. Let's split up between players who stayed on the same team versus those who switched teams. As Eric T. noted, teammates play a role in assists, so numbers on players who change teams might be closer to the truth. So let's see:
n
|
Repeatability
|
A160
|
A260
|
A60
|
1020
|
Forwards-same
|
0.36
|
0.21
|
0.44
|
382
|
Forwards-Diff.
|
0.32
|
0.10
|
0.35
|
Difference
|
-0.04
|
-0.11
|
-0.09
| |
604
|
Defensemen-same
|
0.24
|
0.12
|
0.26
|
232
|
Defensemen-Diff.
|
0.06
|
0.02
|
0.11
|
Difference
|
-0.18
|
-0.10
|
-0.15
|
n
|
Predictivity
|
A160
|
A60
|
1020
|
Forwards-same
|
0.40
|
0.44
|
382
|
Forwards-Diff.
|
0.33
|
0.35
|
1402
|
All Forwards
|
0.39
|
0.43
|
604
|
Defensemen-same
|
0.27
|
0.26
|
232
|
Defensemen-Diff.
|
0.08
|
0.11
|
836
|
All Defensemen
|
0.23
|
0.24
|
As you would imagine total assists edges out first assists in all cases except, oddly, defensemen who stay on the same team. I won't put much thought into that, it's nothing. Also, the edge in each case is really small. I would imagine that this being only one year worth of data contributes to this. As you would imagine, after getting a few years of playing time we can make a better estimate of secondary assists.
Conclusion
The numbers here are, overall, close to the one's shown by Eric T. five years back. And I think the best bet when looking at assists is focusing primarily on primary assists. But secondary assists still matter a little. One year of assists tells us more than primary assists. Not by much, but there is something. And given a few years, we'll get a better judge of a players secondary assist "talent". Just looking at the leaderboards for secondary assist over the past few years will tell you that it means something. Secondary assists may contain a lot of randomness, but they still matter. The gain over primary assists is minimal, but they shouldn't just be discarded as noise (just mostly noise).
**All data courtesy of Corsica.hockey
***Update:
This is a good article on secondary assist-http://fivethirtyeight.com/features/some-nhl-stars-get-more-assists-at-home-than-they-deserve/. The next step would probably be adjusting for rink bias. Also this
is a good chart showing how small the spread of talent is in secondary assists. Even though it would have been better to do a separate chart for forwards and defensemen.@tangotiger SD of z-scores is 1.02, histogram suggests moderate skew/shift right pic.twitter.com/E2bf1jQYCH— Michael Lopez (@StatsbyLopez) April 22, 2016
Wednesday, April 13, 2016
Estimating Shooting Talent
I guess the first question here is, what do I mean when I say shooting talent? That's a fair question. To start off, let's think about shot quality. Shot quality is the probability of a shot going in. This quality is made up of a lot of different things. The distance of the shot from the goal, the angle, the type of shot, if it's a rebound, if it's a rush shot, the score.....etc. Another, sometimes overlooked, component is the the actual shooting talent of the shooter in question. Meaning in hockey parlance, "How good is his shot?". But how do we do this? (Dtmaboutheart's model tried to take this into account in his expected goals model here by regressing sh%, but I disagree with that method)
I think the best way to start to answer this, is to think about what makes up a player's shooting percentage. This as opposed to looking at sh%, because straight sh% isn't exactly shooting talent. A player could have a high sh% because of the quality of shots he takes or because of his actual shooting ability. Nevertheless, I think it's roughly like this:
Observed Sh%= Shot Quality* + Shooter Talent + Randomness
*When shot quality equals to the stuff I talked about in the first paragraph
So what we need to do: Is estimate the shot quality for each player, and regress for randomness and what should be left is our best estimate of a player's shooting talent. And the way to do this, is I think simple. First we'll estimate the quality using the expected goal model developed by Emmanuel Perry over at Corsica.hockey. And then we just do is:
Goals/Expected Goals= Shot Multiplier
Let's think about this. Let's compare two players: Both have the same expected goals but one exceeds in actual goals. So why is one player doing better than the other. Is this because of shooting talent or just randomness? If it's something real we would expect this measure to show some repeatability. If a player consistently scores more goals then we would expect, something is probably going one. But how much? This is obviously a sensitive measurement, so we need to be careful if we are measuring anything real or just pure randomness. So we'll run a regression.
First we have to split up forwards and defensemen. Those are two different positions and we should obviously expect forwards to have more talent here. And I'm going to run three regressions for each group. The first is a regular year over year regression, limited to players who have 500+ TOI in both seasons. But one year for this isn't the best judge. There is bound to be a good deal of randomness here, and it'll probably take a few years to get a better read. So my other two regressions will be 2 years vs. 2 years, and 3 vs. 3. The cutoff being 1000, and 1500 minutes respectively. These numbers are arbitrary and one can play around a little, but it still leaves us with a fine sample size for the analysis. Also there are obviously better ways to conduct this analysis, but this is a quick and easy way to do so. So here you go (the numbers are 5v5 data from 2007-2008- 2015-2016).
** r being the correlation coefficient
So let's see what we got here. First off, on the left in the "n" column is the sample size ordered by forwards then defensemen for that particular regression. What we see is simple and intuitive. There is a good deal of randomness in the data and we see more of a signal for forwards than we do for defensemen. For defensemen, there doesn't seem to be much there. Even after three years it still regresses 87% to the mean. It doesn't mean it doesn't matter at all....but it doesn't seem to make much of a difference (specifically when you take into account the amount of goals you would expect these guys to score anyways).
Forwards are obviously a different story. As you can see, we do get a signal through. There still is a fine amount of noise (especially for one year). But considering, it's not bad. We have to be careful with our data and regress it. Given a few years, we can start to make an estimate of a player's shot multiplier.
That's not it though. There's still one question: What do we regress to? As shown first in this article by Eric T., we can use coaches decisions to find the right mean to regress too. Because players who average more TOI per game tend to shoot higher. Make sense. Coaches know how to evaluate talent, and will therefore give more ice time to players who are better skilled. But we can't just plot TOI/G vs. Sh% (as he notes). I would say most of the players given more ice time will be the more talented one's, but there's also the problem of coaches riding the hot hand. For example, imagine a player with regular shooting talent. But he goes off (like many do) for a portion of one year and starts scoring a ton of goals. It's possible that the coach will then bump him up a line and play him more. So getting lucky and over-exceeding one's talent will therefore be reflected in TOI/G. Because some of those players with high TOI/G will be guys who experienced a decent portion of luck and are now getting "rewarded" with more ice time. So it'll be more extreme then it actually is.
What we should do is plot TOI/G in year 1 vs. Shot Multiplier in year 2. This is because year 2 performance is a better estimation of the players talent. In year two those players who got "lucky" and were given more ice time will on average regress towards the mean, so the bias won't be there anymore (remember this also works the other way with players who got unlucky and were given less ice time). So let's look at it:
This is just for forwards. I'll spare posting the graph for defensemen as it's just a straight line. That's in line with what we know so far. Now this graph for forwards shows a clear upward slope. Those players who who get about 13+ minutes would be expected to regress to higher mean, and those below 13 minutes to a lower one. For any TOI/G all you have to do is stick it in the equation to figure out what we would expect.
Ok so there are a few stuff here, so let's look at a working example. Let's compare two players at the extremes: Steven Stamkos and Tanner Glass over the past three years.
G60 ixG60 Shot Multiplier
Steven Stamkos: 1.23 .86 1.46
Tanner Glass: .23 .40 .60
So Stamkos scores about a goal more per 60, but get knocked down about ~.4 goals per 60 in expected measures. So his multiplier is pretty big. Glass, on the other hand, is projected to score a little under double the amount of goals than he actually did. Resulting in a small multiplier. Let's now regress them.
TOI-Mean Regressed Multiplier
Steven Stamkos: 1.05 1.21
Tanner Glass: .86 .75
So the second column is the mean multiplier we would expect based on the players TOI and should therefore regress to. And the second column regresses their multiplier towards the mean using the numbers discussed in the first chart (it's three years so they are regressed 61% to the mean). Let's look at the final result
Regressed Multiplier x G60
Steven Stamkos: 1.04
Tanner Glass: .30
So the last step is just multiply expected goals by the multiplier. Remember, the multiplier is how much we would expect a player to exceed or fall short of their expected goals. So we would expect Stamkos's G60 to be a higher by a factor of 1.21, and for Glass's to be 75% of his expected G60. I'd like to add that this is only on three year data. If we used players career data and regressed that we would get a better estimate of their shot multiplier. Either way Stamkos here goes up .18 goals per 60 and Glass down .1. These are meaningful differences. These two players are at the extremes so we'll see some of the biggest differences with them, but it still should be looked at.
Conclusion
As expected goal measures become more popular, it's important to remember to "shot talent". Sh% includes both location measures and actual shooting talent. And I would say the best way to measure that in the present time is: Comparing it to results and regressing and using a coach's judgement. It's important to be careful with this, and while it won't capture everything and for a lot of players won't mean much, it's a better way of getting the full picture.
***All Data Courtesy of Corsica.Hockey
I think the best way to start to answer this, is to think about what makes up a player's shooting percentage. This as opposed to looking at sh%, because straight sh% isn't exactly shooting talent. A player could have a high sh% because of the quality of shots he takes or because of his actual shooting ability. Nevertheless, I think it's roughly like this:
Observed Sh%= Shot Quality* + Shooter Talent + Randomness
*When shot quality equals to the stuff I talked about in the first paragraph
So what we need to do: Is estimate the shot quality for each player, and regress for randomness and what should be left is our best estimate of a player's shooting talent. And the way to do this, is I think simple. First we'll estimate the quality using the expected goal model developed by Emmanuel Perry over at Corsica.hockey. And then we just do is:
Goals/Expected Goals= Shot Multiplier
Let's think about this. Let's compare two players: Both have the same expected goals but one exceeds in actual goals. So why is one player doing better than the other. Is this because of shooting talent or just randomness? If it's something real we would expect this measure to show some repeatability. If a player consistently scores more goals then we would expect, something is probably going one. But how much? This is obviously a sensitive measurement, so we need to be careful if we are measuring anything real or just pure randomness. So we'll run a regression.
First we have to split up forwards and defensemen. Those are two different positions and we should obviously expect forwards to have more talent here. And I'm going to run three regressions for each group. The first is a regular year over year regression, limited to players who have 500+ TOI in both seasons. But one year for this isn't the best judge. There is bound to be a good deal of randomness here, and it'll probably take a few years to get a better read. So my other two regressions will be 2 years vs. 2 years, and 3 vs. 3. The cutoff being 1000, and 1500 minutes respectively. These numbers are arbitrary and one can play around a little, but it still leaves us with a fine sample size for the analysis. Also there are obviously better ways to conduct this analysis, but this is a quick and easy way to do so. So here you go (the numbers are 5v5 data from 2007-2008- 2015-2016).
n
|
TOI
|
Forwards (r)**
|
Defensemen (r)**
|
1974/1213
|
500+
|
.17
|
.04
|
1333/784
|
1000+
|
.30
|
.10
|
782/452
|
1500+
|
.39
|
.13
|
** r being the correlation coefficient
So let's see what we got here. First off, on the left in the "n" column is the sample size ordered by forwards then defensemen for that particular regression. What we see is simple and intuitive. There is a good deal of randomness in the data and we see more of a signal for forwards than we do for defensemen. For defensemen, there doesn't seem to be much there. Even after three years it still regresses 87% to the mean. It doesn't mean it doesn't matter at all....but it doesn't seem to make much of a difference (specifically when you take into account the amount of goals you would expect these guys to score anyways).
Forwards are obviously a different story. As you can see, we do get a signal through. There still is a fine amount of noise (especially for one year). But considering, it's not bad. We have to be careful with our data and regress it. Given a few years, we can start to make an estimate of a player's shot multiplier.
That's not it though. There's still one question: What do we regress to? As shown first in this article by Eric T., we can use coaches decisions to find the right mean to regress too. Because players who average more TOI per game tend to shoot higher. Make sense. Coaches know how to evaluate talent, and will therefore give more ice time to players who are better skilled. But we can't just plot TOI/G vs. Sh% (as he notes). I would say most of the players given more ice time will be the more talented one's, but there's also the problem of coaches riding the hot hand. For example, imagine a player with regular shooting talent. But he goes off (like many do) for a portion of one year and starts scoring a ton of goals. It's possible that the coach will then bump him up a line and play him more. So getting lucky and over-exceeding one's talent will therefore be reflected in TOI/G. Because some of those players with high TOI/G will be guys who experienced a decent portion of luck and are now getting "rewarded" with more ice time. So it'll be more extreme then it actually is.
What we should do is plot TOI/G in year 1 vs. Shot Multiplier in year 2. This is because year 2 performance is a better estimation of the players talent. In year two those players who got "lucky" and were given more ice time will on average regress towards the mean, so the bias won't be there anymore (remember this also works the other way with players who got unlucky and were given less ice time). So let's look at it:
This is just for forwards. I'll spare posting the graph for defensemen as it's just a straight line. That's in line with what we know so far. Now this graph for forwards shows a clear upward slope. Those players who who get about 13+ minutes would be expected to regress to higher mean, and those below 13 minutes to a lower one. For any TOI/G all you have to do is stick it in the equation to figure out what we would expect.
Ok so there are a few stuff here, so let's look at a working example. Let's compare two players at the extremes: Steven Stamkos and Tanner Glass over the past three years.
G60 ixG60 Shot Multiplier
Steven Stamkos: 1.23 .86 1.46
Tanner Glass: .23 .40 .60
So Stamkos scores about a goal more per 60, but get knocked down about ~.4 goals per 60 in expected measures. So his multiplier is pretty big. Glass, on the other hand, is projected to score a little under double the amount of goals than he actually did. Resulting in a small multiplier. Let's now regress them.
TOI-Mean Regressed Multiplier
Steven Stamkos: 1.05 1.21
Tanner Glass: .86 .75
So the second column is the mean multiplier we would expect based on the players TOI and should therefore regress to. And the second column regresses their multiplier towards the mean using the numbers discussed in the first chart (it's three years so they are regressed 61% to the mean). Let's look at the final result
Regressed Multiplier x G60
Steven Stamkos: 1.04
Tanner Glass: .30
So the last step is just multiply expected goals by the multiplier. Remember, the multiplier is how much we would expect a player to exceed or fall short of their expected goals. So we would expect Stamkos's G60 to be a higher by a factor of 1.21, and for Glass's to be 75% of his expected G60. I'd like to add that this is only on three year data. If we used players career data and regressed that we would get a better estimate of their shot multiplier. Either way Stamkos here goes up .18 goals per 60 and Glass down .1. These are meaningful differences. These two players are at the extremes so we'll see some of the biggest differences with them, but it still should be looked at.
Conclusion
As expected goal measures become more popular, it's important to remember to "shot talent". Sh% includes both location measures and actual shooting talent. And I would say the best way to measure that in the present time is: Comparing it to results and regressing and using a coach's judgement. It's important to be careful with this, and while it won't capture everything and for a lot of players won't mean much, it's a better way of getting the full picture.
***All Data Courtesy of Corsica.Hockey
Subscribe to:
Posts (Atom)