THE MYTHS WE LIVE BY: THE DATA WE USE AND ITS LIMITATIONS By Bill Oppenheim

Editor=s note: Bill Oppenheim delivered this speech at
the Pedigree/Genetics/Performance Conference in
Lexington, KY, September 7, and it is being reprinted
here with the permission of the organizers.
From TDN 21st September.

Most of you know me as a columnist for the
Thoroughbred Daily News, so I'd like to briefly outline
my credentials to speak here today. When people ask
me what I do, I tell them, "I'm an analyst in the
breeding and auction sales sector of the Thoroughbred
horse industry." The usual reaction to that is, "say
what?"
As an analyst, one of my jobs is a journalist, and
that's how I'm best known. But another of my jobs is as
a pedigree consultant, advising on matings and
purchases at sales. I've also done this job for about 30
years, ever since John Ed Anthony hired me to help him
select mares to go to Cox's Ridge when that horse
retired to Claiborne Farm, I think in 1980. I would
advise on perhaps 200-300 matings a year and have
been fortunate to have been involved with three major
operations over that time, which means the advisor is
working with at least some top mares and sires, which
of course enhances your chances. I've been involved in
advising on the matings of 13 Classic winners since
1992, including two winners of the Epsom Derby, three
consecutive European champion 2-year-olds, two Arc
winners, and the winners of the Epsom Oaks in 2007,
2008, and 2010. I report this simply to say this is a
field I have worked in for 30 years with some success,
so I do bring some practical experience to the table.
I also want to thank two people who have been
instrumental in putting together this presentation (I'm
very low-tech, myself): my now long-standing (I almost
said long-suffering) associate of five years, Brianne
Stanley; and Dr. Emily Plant, an Assistant Professor at
the University of Montana, who reviewed, critiqued, and
helped us assemble the presentation.
SECTION 1. THE CONDITIONS WE WORK WITHIN
There are a few things I really like about how the
Thoroughbred business, especially as what I call the
breeding and auction sales sector of it (as distinct from
the racing sector) has evolved. I like that it is a truly
international business, a 'business without borders;'
sure, there are a lot of wholly national characteristics of
the industry in each country, but when we talk about,
say, bloodstock agents who travel to and buy at the
major international auctions, we mean Kentucky,
Newmarket, Goffs in Ireland, Deauville, at least, but the
buyers are coming from Australia, Japan, the Middle
East, South America, South Africa, Russia, India, Korea,
Turkey, and many other countries and regions as well.
In a very important respect, the 'breeding and sales
sector,', anyways is a business without borders.
Second, we are fortunate to be working in an
essentially unregulated 'free market' economy. Of
course, there are rules and regulations, but there are not
artificial, externally imposed barriers to, or conditions of,
trade. It makes it more feasible to analyze how certain
principles operate, such as one of the most important--
the law of supply and demand. Also, this has become a
big-money business. Thoroughbred horses can be worth
really big money, so much so that it has created its own
economy. This is because Thoroughbreds are not just
racehorses and breeding stock; Thoroughbred horses are
also financial instruments, like, say, commodity options,
or derivatives. The difference is these financial
instruments are live animals, too.
Third, at the end of the day-- I mean, at the end of
the race--there is a finish line, and a result. Horse racing
itself has to be a meritocracy--the best horse wins. As
an aside, that's what I personally dislike about handicap
racing, because the intention is for the best horse not to
win, but, fortunately, in the really important races from
the long-term perspective of the breeding standpoint,
the best horse wins. And, fortunately for many of us, in
the last 30 years, the business itself has become much
more of a meritocracy. If you have the best idea, you
will be heard, maybe even funded. And the wealth in
the business is spread out much more democratically; if
you come up with the right horse, you will get paid, no
matter who you are, because the best horse wins.
That, as I see it, is the essential landscape we are
working in: these are the conditions in which we all
operate--and I, for one, am grateful that we can work in
a business, a real business, too, in which you don't
have to put a tie on to go to work every day. Being a
big-money business, there is a fair amount of greed and
chicanery to be found, but there is also a very serious
demand for advice and assistance in the creation or
discovery of the superior Thoroughbred. That's where
we all--the speakers at this symposium--come in.
SECTION 2. WHAT WE DO
So far today you've listened to a fair bit of science,
but this afternoon I'd like to change the focus a little bit,
to what we might call more traditional methods of
looking at pedigrees, and deciding on matings. When I
refer, in the title of my talk, to "the limitations of the
data that we use," I'm talking more about us boffins
with our books than the data the scientists use; I'll have
to leave them to comment on their data, that issue isn't
one I'm really competent to comment on.
There is one assumption we share, though, and I
emphasize the word "assumption" because I think it's
vital that we understand the assumptions from which
our research and conclusions emanate. And the
assumption that scientists and more traditional pedigree
students share is this: we want to replicate the
pedigrees of good horses. Students of DNA will look to
replicate the patterns they find in top horses; pedigree
students looks for patterns in the pedigrees of good
horses. But we're all working from the same
assumption: that which has happened before is more
likely to happen again than that which has not happened
before; we're trying to replicate past successes,
however we define those successes. The $64-million
dollar question is: what is it that we're trying to
replicate; what has succeeded before?
Let me cite three examples which I think challenge
some of the conventional wisdom which prevails among
pedigree students today.
First there is the APitfall of the Small Sample Size,@ in
which a few early successes lead breeders to flock to
the hot new cross, only to discover the law of averages
tends to even things out the more occurrences there
are. The greatest example I have ever seen of this was
the case of Storm Bird over Secretariat mares. Storm
Bird has two named foals out of Secretariat mares in his
first crop, in 1983, and they were both graded-stakes
winners: Grade I winner Storm Cat, and a Grade
III-winning filly Storm Star. The crops weren't so big
then as they are now, as you know, and by the 1987
crop, there had been only seven more named foals by
Storm Bird out of Secretariat mares. One of the four
foals of 1987, though, was Summer Squall; in 1991
there were 5 named foals bred on this cross, and in the
next four years there were 9, 9, 16, and 12,
respectively.
If you'd taken a snapshot of Storm Bird over
Secretariat after his first eight crops--and evidently
plenty did--you'd have made the staggering discovery
that the cross of Storm Bird out of Secretariat mares
had thus far produced 36 percent stakes winners, and
29% graded-stakes winners. What would you have
done if you had a Secretariat mare?
Exactly; in his last 10 crops, Storm Bird had 60
named foals out of Secretariat mares. They included one
black-type winner, and no graded SW.
First 8 crops:36%; last 10 crops, 2%.
Why? Did the cross suddenly become less potent?
No. Breeders were fooled by a few good early results
from a sample size which was of no statistical
significance. Anything could have happened. When it
was all totaled up, Storm Bird sired 9% black-type
winners from foals, and 5% graded SW from foals. Out
of Secretariat mares, his career totals were 8% and
5%, respectively; so really, it was just an average cross
for the sire, which happened to have all its stars arrive
in the first half of his career.
So, beware of the small sample size, I say. Here's
another nugget of conventional wisdom which the facts
disprove, and which actually applies to the Storm Bird--
Secretariat example as well. Conventional wisdom
believes when you find a winning formula, a pattern
that works, you keep repeating it; but, I maintain, as
soon as we know something works, it is outdated; it is
on the way out. Here's why: there is a four- or five-year
lag between when we decide on a mating and the
resulting produce gets to the racetrack. In that time
there can be significant shifts in the population; certain
lines become numerically stronger or weaker within the
population. A line which has become stronger was
probably under-represented proportionally five years
ago, and vice versa. When you did a mating five years
ago, you were working with a different population.
Here's another example of a cross which was
outdated the minute it was discovered: Sadler's Wells
over Darshaan. There were no foals by Sadler's Wells
out of Darshaan mares until 1992; both had their first
foals the same year, 1986. There were two graded SWs
bred on that cross in 1997 from his 1994 crop, Crimson
Tide and Ebadiyla. From the 1998 crop came three
black-type winners from five foals out of Darshaan
mares, including 2001 St. Leger winner Milan; and out
of the eight named foals by Sadler's Wells out of
Darshaan mares in the 1999 crop came High Chaparral,
Islington, and Quarter Moon in 2002.
From the 12 crops foaled from 1992-2003, there
were a total of 66 foals bred on this cross. Seventeen
of them (27.5%) were black-type winners, 12 of those
(18.2%) group winners.
Once it was obvious this was a very powerful cross
(by 2003), the numbers jumped up, and the strike-rate
declined. In his five crops foaled between 2004-2008,
there were a further 97 named foals bred on this cross.
The numbers weren't terrible--11.3% black-type
winners, 8.2% group winners--but the cross worked
less than half as well after it was clear this was a
sensational cross than it had before.
These are myths, these conventional wisdoms that
we live by. It is a myth that a small number of good
horses from an insignificant sample is a reliable
predictor. It is a myth that once we discover a great
cross, it will continue to be just as great a cross as it
was before we discovered it. These myths crash on the
rocks of the laws of probability; in these cases, the
greater the numbers, the less the success.
The third myth I should like to address has to do with
the question: "what class does a horse need to be
before we want to replicate its pedigree?" This is an
absolutely essential question. The further you go down
the class scale, the more 'random' the pedigrees are.
Conventional wisdom is that the pedigrees of all blacktype
winners are worth collecting and studying. Jack
Werk used all black-type winners when he created his
database--I believe that is correct--and True Nicks has
followed suit, introducing the supposed 'improvement'
of being able to measure opportunity. I believe I am
right about that, but please correct me if I'm wrong
[correction from the audience: he did not use restricted
black-type winners]. But it sounds quite sensible,
doesn't it? All black-type winners.
The trouble is, according to my research, all blacktype
ain't equal. We have plenty of anecdotal evidence
of this in the number of Europeans who come to the
American sales and stare blankly at the catalogue
pages, trying to figure out what in the hell does all this
black-type really mean? There's a reason why.
In Europe, there are two kinds of black-type races:
group races, and listed races. But in North America
there are three kinds: graded races--same as group;
listed races; and 'other' black-type races--I think, now,
these are any black-type races with a gross below
$75,000. Yes, the percentage of black-type races in
Europe is higher than in North America--though, in
2008, the number of races run in Britain, Ireland, and
France combined was less than a quarter of the number
of races run in North America.
In any case, it sort of works. I think it would make a
lot more sense to call listed races Grade 4, or Group 4,
because that's really what they are. But here's a vital
statistic: in 2008, there were 1,890 black-type races in
North America, according to The Jockey Club Fact
Book. Of those, 742, or 39%, were listed or graded
races, and 1,148, or 61%, were not. In Britain, Ireland,
and France, in the same year, there were 630 blacktype
races; none were below listed class. 61%; 0%:
something's out of whack.
As some of you may know, I use a somewhat
different way of identifying class, which is earningsbased,
and identifies a group called 'A Runners,= which
are the top 2% of earners in designated countries each
year. I mention this only to say I once did a study which
told me the percentage of A Runners in the population is
very close to the percentage of listed plus graded and
group winners.
The point is, I have learned, using my method, that
only the pedigrees of A Runners are worth studying; the
pedigrees of B Runners, C Runners, everything else, are
increasingly random. So I infer that black-type winners
below listed standard are the equivalent of B Runners
and C Runners in my system: not worth studying.
My conclusion: if we look at the graph here on Pitfall
#3, we can see that, in 2008, 45% of the black-type
winners in the five countries combined were below
listed class; in other words, in my opinion, nearly half
these databases are junked up with horses which
shouldn't be there. I'd say it's pretty likely the
conclusions could be riddled with 'false positives.'
Byron Rogers, one of the organizers of this
conference, is a very smart guy--one of the smartest
I've met in this business. One of the big selling points of
TrueNicks over Jack Werk's material, I remember him
telling me, was their ability to "measure opportunity"--
not just has something worked (given my caveat about
both groups' databases), but how many times has it
really worked compared to how many times it's been
tried?
It sounds good on paper, but the more I've thought
about it, the less I like it. There's eminent danger of the
small sample size problem in almost any case--oh, this
has been tried 16 times, this has been tried 24 times,
and so on. When you say, well, something's better than
nothing, you have to understand, in these cases, no, it's
not.
Somewhere between 2 and 3% of horses, depending
on the study and the class parameters used, are actually
good horses, horses the pedigrees of which are worth
trying to replicate. Let's call it 2.5%--one out of 40. So
39 out of 40 fail to make the cut. With such an
incredibly high failure rate, I really have to question
whether measuring opportunity can be truly significant.
Most horses fail--the vast majority of horses fail. If
something works three out of 40 times, and something
else works one out of 40 times, does that mean we
should we doing the three-out-of-40 every time? Maybe
not.
Then, if you want to go one step up in the pedigree;
in other words, not sire on damsire, but something like
grandsire and sons on damsire, and so forth, another
problem arises: where do you make the break? Is there
such a thing as a sire line, and if so, where do you
break it? Ribot was always Ribot, even if it is Albertus
Maximus, by Albert the Great, by Go for Gin, by
Cormorant, by His Majesty, by Ribot; five generations
back. The identity and classification of sire lines is
subjective, not objective; everybody is eligible to use
something different.
Measuring opportunity could be significant if you were
looking at enough cases; but how should they be
properly grouped? There's no agreement on this. All
taken together--I used to be quite interested in
measuring opportunity. I'm not at all any more; I'm
really much more interested in accurately identifying
success.
SECTION 3. THE IMPLICATIONS OF UNCERTAINTY
Every time a breeder signs on the dotted line to breed
his or her mare to Distorted Humor, or Unbridled's
Song, or Giant's Causeway--and a few others--it's near
enough a $100,000 decision. It should make a
difference to those of us who work in the field of
evaluating and recommending pedigrees, where
percentage plays are so important, that we are speaking
from a vantage point of uncertainty, not of certainty.
Even from the point of view of the scientists, we're
entitled to ask "how do you even know what to
measure?" and "how can we know that what you do
measure is significant?" Whatever we do, we're taking
a shot here: the odds in any given case are against
success.
It bothers me when I hear someone assert "this or
that is absolutely the way to go;" it bothers me when I
say it myself. At best it is absolutely the way to go,
given the circumstances. I try not to be too militant,
above all to be open-minded, to use inductive as well as
deductive reason--let the evidence tell you, not the
other way around.
An example is the application of dosage to matings
planning. Dr. Roman, who will be speaking tomorrow,
will be able to explain it much better than me. I don't
have a problem with the theory of dosage, but with the
application of it. Here's what I mean. As I understand it,
dosage is a sort of pedigree map, showing the important
names in pedigrees and the aptitudes of those named
sires. A calculation is then done which shows the
distribution of influences in the existing pedigree, from
which the idea is that you can predict aptitude, and add
certain elements to balance out the pedigree; in some
way to produce a good horse.
It seems to me the problem here is an assumption
the assumption that the individual horse on the ground,
or the prospective individual horse on the ground will
resemble the designed pedigree. My experience is that
most horses throw back to certain influences in their
pedigrees, except, given the genetic law that
populations tend to breed back to the middle, you might
find a horse "throws back" to a certain sire, but isn't as
good.
If horses throw back to particular influences in their
pedigrees, surely a map of a horse's dosage tells us
simply about the opportunity, not the fact, of influence.
Like I say, my issue is not with the theory of dosage,
but the application of it: the pedigree may be that of a
two-mile stayer, but if the horse on the ground looks
like a sprinter and stops after five furlongs, I'd be
looking for the names in that pedigree of five-furlong
horses, and start working from the assumption that's
the horse we're really dealing with.
There are two corollaries to approaching pedigree
assessment with an open mind: one is the recognition
that, even in science, the development of hypotheses
requires a creative 'spark'--somebody has to think up
the right question to ask, the right hypothesis to test.
The other corollary is what is called to "think outside
the box.@ Let me give you an example that utilizes both.
At the end of 1985, I was hired to make
recommendations for the American mares of one of the
major international operations. There was a particular
mare, a Halo mare, who I was keen to see go to Mr.
Prospector. His first foals had raced in 1978, but until
then--1985--he had yet to sire a good horse out of a
mare from the Hail to Reason sire line. But, expanding
the parameters one level, I knew that Mr. Prospector's
sire, Raise a Native--and, in fact, virtually all Native
Dancer--really liked Hail to Reason-line mares. So I
explained the situation, explained why I thought it
would still be worth trying, and they rolled the dice.
They got Machiavellian. From then on, I became a lot
less rigid about insisting that this sire had to have
worked with that damsire.
The point I'm ultimately driving at is that we are
operating in a universe, this universe of Thoroughbred
racing and breeding, which is more chaotic than
ordered. There is a natural tendency to want things to
make sense and be in order. But that path leads to
overrating the true, applicable value of the data we are
using and the conclusions we reach from that data. It's
the search for the one great formula, or set of formulas:
the Philosopher's Stone, the Magic Wand, the Secret
Elixir. I don't think they exist, not in this pursuit,
breeding racehorses.
Then what?
Then, we have to recognize--to concede--that the
pedigree work is only part of the equation; it's
dangerous to make all this that we discuss too big a
factor in the decision-making process; at the end of the
day, it's an individual bred to an individual. Let me just
reiterate the reservations I have about the assumptions
which result in conventional wisdom:
1. Jumping to conclusions based on too small a
sample size;
2. By the time we discover a particular cross is really
good, it is losing its potency;
3. The assumption that all black-type should count the
same as an indicator of class is flawed;
4. The value of measuring opportunity is open to
question;
5. A pedigree is a map of opportunity, not a statement
of fact. Horses are not blends;
6. Generalizing 'up one level' is often more productive
than trying to be too specific;
A few final thoughts: the first thing we have to do is:
eliminate obvious errors. Then, get ahead of the curve;
and do art as well as science. Finally--it's blindingly
obvious that the people who use the services we're all
working to develop want the silver bullet, the holy grail,
the magic formula. Yes, let's use all the tools we have
and can develop, but let's also try very hard to make
them understand, as we should, the limitations of what
we are doing. Thank you.