We recently expanded our database of round scores to include a
comprehensive set of amateur events dating back to the fall of 2009.
This includes any event that is eligible for points in the World Amateur
Golf Rankings (WAGR), as well as the vast majority of college
events played in the United
States (most of which are also included in the WAGR). By linking the golfers
in the amateur data to our existing data for the professional tours, we are able
to estimate a single strokes-gained measure that can be directly compared
across any tournament and tour. That is, a value of +2 for this measure
for a round played in the Canadian Men's Amateur and a value of +2 for a round
played in the U.S. Open will indicate performances of equal quality. Pretty cool;
but why should you believe our numbers? The purpose of this blog is to provide
some transparent arguments for the validity and usefulness of this adjusted,
or "true", strokes-gained measure.
We'll tackle validity first. The motivating idea behind converting raw scores to
adjusted scores is that we only want to compare golfers who played at the same
course on the same day, in order to control for the large differences in course difficulty
that exist across the many global tours and tournaments. However, the obvious issue
is that most golfers
never compete directly against each other.
To compare scores shot in different tournaments, we make use of the fact
that there is overlap in the set of golfers that compete in different tournaments.
For example, if Matthew Wolff beats Justin Suh by 2 strokes per round at the NCAA
Championship, and then Rory McIlroy beats Matthew Wolff by 2 strokes per round at a PGA
Tour event a few weeks later, we could (maybe?) conclude that McIlroy is 4 strokes per round better
than Suh, despite the fact that they never played against each other! Of course,
this seems like a bad idea because we know that golfers have "good" and "bad" days;
if Wolff had a good day when he played against Suh, and a bad day when he played
against McIlroy, this would lead us to overestimate how much better McIlroy is than Suh.
This problem is mitigated if we have not just one golfer connecting McIlroy to Suh but many:
some golfers will have bad days and some will have good days, but on average it should
even out. The basic logic described here allows us to fairly compare scores shot in
major championships to those shot in tournaments as obscure as the Slovenian National Junior
Championship. All that is required is that these tournaments can be "connected" in some way;
that is, a player in the Slovenian Championship needs to be able to say, "I played against a golfer who played against a golfer... who played against a golfer in a major
championship". The fewer connections a tournament has to the rest of the golfing world, the less
certain we can be about the quality of the performances from that tournament; in the extreme case
of having no connections, we can say nothing about how those performances compare to other tournaments.
Therefore, to be included in our database, all tournaments must be connected in this sense. Luckily, competitive golf,
like the real world
is a lot more connected than you might think.
Let's next look at some simple data summaries to provide some reassuring evidence that
the true strokes-gained metric is accomplishing what we want it to.
This first table compares a golfer's performance in Korn Ferry Tour events
in 2019 to their performance in PGA Tour events in 2019. We show 10 golfers who played
a reasonable number of rounds on both tours last year:
The far right column is the one of interest: this tells us how many more strokes
per round each golfer gained against KFT fields than they did against PGA Tour fields. A positive
value indicates that they beat KFT fields by more on average; a negative value
indicates the opposite. Why would the same golfer beat KFT fields by more than
PGA Tour fields? Presumably, it's because the average player on the latter tour is
better than the average player on the former. The fact that there are large differences between golfers in
their relative performance on the two tours simply reflects the fact that golf
performance is random; some golfers may happen to play really well
in the PGA Tour events they enter (e.g. Lanto Griffin).
The next table shows the overall numbers from this exercise
for 2019 for 3 different tours:
Looking at the first row, we see that in total there were 2095 "shared" rounds —
defined for each golfer as the minimum of the number of rounds they played on
each tour, e.g. for Sucher in the first table
it would be 38 — and that on average the same golfer
gained 0.69 more strokes
per round against KFT fields than they did against PGA Tour fields. The final column is our estimate,
from the true strokes-gained method, of the difference in the average player quality
KFT and PGA Tour fields used in this calculation. Ideally we want these last two
columns to be equal — if the average golfer is in fact 0.72 strokes better in these
PGA Tour events, we should see a given set of golfers gaining approximately 0.72 fewer strokes
against those fields, which we do.
The next two tables repeat the exercise using golfers who played in amateur events in
In this second set of tables, we see that
when amateurs play in PGA Tour events the difference between their
strokes-gained over fields on the PGA Tour and their strokes-gained over amateur fields
is on average equal to 3.7. From this we should infer that the average
golfer in these PGA Tour events is 3.7 strokes better than the average golfer in these
amateur events; almost exactly (3.61) what our true strokes-gained method estimated the difference
Adjusted or "true" strokes-gained
is equal to a golfer’s strokes-gained over the field plus
how many strokes better
(or worse) that field is than some benchmark
. Because it is trivial
to calculate strokes-gained over the field, it should be clear that the validity of true strokes-gained
hinges on properly estimating field strengths. In the exercise above we showed
that our estimates of average player quality for different tours closely match the
estimates you would obtain by simply comparing the performance of a set of golfers
against one tour to the performance of that same set of golfers against
another tour. This provides an intuitive way to understand differences in field strength
across tours rather than having to simply place blind trust in our model being correct.
What is this benchmark I mentioned above?
Strokes-gained is a relative measure, meaning that it is only useful insofar
as we have something to compare it to. Just as it’s not very useful to tell
me the raw score of a golfer (e.g. 66) without providing the field’s average
score that day, it’s also not useful to tell me that a golfer gained 4 strokes
without telling me who they "gained strokes over".
We typically set the benchmark to be the average field on the PGA Tour in a given season.
Therefore, if, in the year 2019, a golfer has an adjusted strokes-gained of +2 in a given round,
this means that their performance was estimated to be two strokes better
than what the average PGA Tour field in 2019 would be expected to
shoot on that course. An adjusted strokes-gained of +2 could be achieved
by beating a field that is on average 1 shot better than the average
PGA Tour field (e.g. the TOUR Championship) by 1 stroke, or by beating a
field that is on average 1 stroke worse than the average PGA Tour
field (e.g. a typical Korn Ferry Tour event) by 3 strokes.
Hopefully this has helped clarify how the true strokes-gained
measure is derived and also why it is valid to directly compare the true strokes-gained
numbers of golfers who competed in different tournaments. Next, we are going to show why
the true strokes-gained metric is useful
The World Amateur Golf Rankings
are put on by
the USGA and R&A with the stated aim
of providing a comprehensive and accurate ranking of amateur golfers. We are going to compare our
amateur ranking system
, which is solely based
off the true strokes-gained metric, to the WAGR system. More precisely, our amateur rankings are determined
by taking a weighted average of each golfers' historical true strokes-gained, with more recent rounds receiving more weight.
For golfers with many rounds played, approximately their most recent 150 rounds contribute to their ranking. The WAGR operates in
a similar fashion to the Official World Golf Ranking
(OWGR) with rounds from the
last 104 weeks contributing to their rankings. The main differences
between our rankings and the WAGR is that we only include stroke play rounds while the WAGR includes both stroke
and match play, and that the WAGR rewards top finishers disproportionately — as all official rankings systems do,
and should — while our rankings only depend on golfers' scores and not their finish position.
The exercise we will do is the following: first, we form our rankings for every week since September 2011 using the same
time frame as WAGR; that is, for a given week all events that have been played on or before
the most recent Sunday are included in the ranking calculation. (WAGR
releases rankings on Wednesday, so events that are played on the Monday and Tuesday of that week only count
in the following week's rankings.) Second, we use each of these ranking systems to predict the performance
of golfers who are playing in the following week
. To be clear, the events we are predicting
always occurred after the date on which the rankings were formed.
Our measure of performance is strokes-gained relative-to-the-field — this is what
we are trying to predict. An example will help clarify
exactly how this prediction exercise works. In the first round of the
2020 Junior Invitational at Sage Valley, which took place on March 12-14, there were 37
golfers that could be matched to both the WAGR and our rankings on March 11, 2020. The field was comprised of 52 golfers,
meaning that 15 golfers weren't matched either because they hadn't played a round in the last year (requirement to be in
our rankings), or they weren't in the top 1000 of the WAGR. Using these 37 golfers, we subtract the average value of our
3 variables of interest: their round score, their Data Golf (DG) ranking, and their WAGR ranking. Then, we correlate their
with their score relative to the field. For example, heading into this tournament Conor Gough was ranked
27th in the WAGR and 279th in the DG rankings; Gough ultimately shot 72 in the first round. The average WAGR ranking of the
36 golfers playing in this first round was 372, and their average DG ranking was 271; their average score that day
was 73.3. Therefore, Gough was ranked well above average in this field
by WAGR (+345), was ranked slightly below average by DG (-8), and he ended up beating the field by 1.3 strokes (+1.3 strokes-gained).
Therefore, in this single instance,
the WAGR rankings could be said
to have been more predictive of Gough's performance that day.
We do this exercise using every tournament-round in our database where at least 2 golfers can be matched to both the WAGR and our rankings.
This results in 306,431 data points to predict. Finally, we run 3 regressions (simply think of a regression as a way of predicting some
variable Y with some other variable X): 1) a regression of a golfer's strokes-gained on their DG ranking; 2) a regression of their strokes-gained
on their WAGR ranking; and 3) a regression of strokes-gained on both their DG and WAGR rankings. The summary measure
we will use from these regressions is the percentage of strokes-gained that can be explained
by each ranking system (e.g. if
the DG rankings indicate golfer A is ranked higher than golfer B, and golfer A goes on to beat golfer B, we would say the DG rankings
"explained" this result to some degree). In statistical parlance this is known as the "R-squared" of a regression.
Finally, let's see some results. When using the previous week's ranking to predict performance in the current week, the WAGR
can explain 4.93% of golfers' strokes-gained; the DG rankings can explain 6.67%; and using the rankings together they
can explain 6.72%. (The low values for these percentages reflects the fact that golf performance on any given day
is mostly not explainable by past performance.) If we perform the same exercise described so far, except using the rankings from 2 weeks ago, or 3 weeks ago,
etc, we get the following results:
As you would expect, the predictive power for both ranking systems declines as we predict further into the future.
For any time horizon, it can be seen that the DG rankings vastly outperform the WAGR in terms of predicting performance.
The "both" line represents the quality of predictions that can be achieved by leveraging information from both ranking systems; for short-term
predictions, WAGR does add a bit of predictive value to the DG rankings, but for longer-term predictions this help disappears.
The small added benefit of WAGR likely comes from the fact that there is some value in incorporating match play performances into amateur rankings,
and also because there were some events where we could not apply our scoring adjustment method and thus do not contribute to the DG rankings.
The fact that using both DG and WAGR together adds very little value on top of DG alone indicates that if you know
a golfer's DG ranking, additionally knowing their WAGR rank provides only a small amount of new information with regards to the quality
of the golfer.
These results should not be surprising; we constructed our rankings to predict performance as well as possible. Conversely, the
WAGR also has to consider notions of fairness and deservedness: if, in the U.S. Amateur, a golfer plays a
mediocre first two rounds but qualifies for the match play portion and ultimately wins the tournament, they will
receive a huge boost to their WAGR position. However, their standing in our rankings may hardly rise, or may even decline
depending on their previous skill level, because we only take into account the stroke play portion of the U.S. Amateur.
Even though match play rounds have little predictive power (not zero; but not a lot) for a golfer's future performance,
it is still likely the case that a ranking system should
reward top performances in match play events. More generally, the WAGR
rewards wins and top finishes disproportionately — the difference in WAGR points between finishing 1st and 2nd is larger
than the difference between 10th and 11th. Again, this is not ideal for maximizing the predictive power of your rankings,
but in our opinion it is how an official ranking system should function. All of this is just to say that the main reason
the DG rankings outperform the WAGR in predicting future performance is that our rankings were constructed with that as the
sole objective, while the WAGR has other factors to consider. Finally, for inquiring minds, the correlation between
the DG and WAGR rankings is equal to 0.76 — meaning that, for the most part, the two ranking systems do generally agree.
To conclude, let's look at one dimension of the WAGR where an undesirable bias could exist: that
tournaments played in specific geographic regions of the world recieve too many points on average. This is a logical
dimension to look at because there is less overlap between the set of golfers competing at
tournaments in distinct parts of the globe, making it more
difficult for the WAGR point allocation to accurately reflect field strength differences.
Unfortunately, we don't actually have
exact data on where tournaments were played, but as a proxy for that we use a golfers' nationality; below
we plot golfers' WAGR rank against their DG rank for a few different nationalities:
These plots use end-of-year WAGR and DG rankings for the years 2011, 2013, 2015, 2017, and 2019. (Even years
are excluded so we don't double count, as the WAGR operates on a 2-year cycle). Each data point represents a golfer
in one of those years. The plots with a majority of data points above the 45 degree line (i.e. DG = WAGR) indicate that
golfers from that country are, on average, ranked worse in the WAGR than the DG rankings; plots with a greater number of points
below the 45 degree line indicate the opposite — that the WAGR ranks them more
favourably than our rankings.
The United States, Japan, and Korea all fall into the former category, while Continental Europe, Australia, New Zealand, and
South Africa fall into the latter. Golfers from the UK and Ireland are ranked similarly on average by the two ranking systems.
Given that golfers don't just compete in their home countries — the most obvious example of this
being the US college golf scene — our guess is that these plots underestimate the bias present in the tournaments that actually
take place in the listed countries. That is, even though most European amateurs play some of their events in the United States, the fact
that they play a higher-than-typical fraction in Europe is enough to introduce the bias we see in their plot. Our finding that
golfers from Japan and Korea are treated unfavourably
by WAGR is surprising, as we
have found the opposite to be true
in the Official World Golf Rankings. The finding that players in the United States are ranked worse in the WAGR
than in our rankings is similar to what we have found when analyzing the OWGR, and also with
that Mark Broadie has done
regarding bias in the OWGR.
There are many other dimensions that could be explored to better understand any inconsistencies and biases in the WAGR system.
With our strokes-gained model, it is possible to drill down to the event level and examine whether too many or too few
points are being allocated to a given event. The basic idea is that we are able to estimate the expected WAGR points
for a golfer of a given
skill level — e.g. the average NCAA D1 golfer — at each event; the higher the expected number of WAGR points for this golfer,
the more favourable is the WAGR point allocation at that event. That will have to be explored another day, however, as I am tired
and this blog is already too long.