Diamond Future’s Do-It-Yourself column is a periodic look at the methods and philosophy that we employ to uncover and evaluate baseball prospects. Over the course of time we will detail both our methods and mindset, which will hopefully give you a different perspective and an opportunity to utilize some of these techniques to improve your own evaluations. Out inaugural article focuses on an overview of our process but it also takes a look at the long-standing conflict that exists between ‘scouts’ and ‘statheads’ and hopefully provides you with an understanding of where we fall on this issue.
For over a decade now there has been this on-going feud within the baseball community between the scouting organizations and the analytical community that was spawned by the work of people like Bill James, Pete Palmer and John Thorn. It came to somewhat of a boiling point in 2003 with the publication of Michael Lewis’ look at Billy Beane and the Oakland A’s in Moneyball. Then when the Theo Epstein led Boston Red Sox won the 2004 World Series, the three decade old sabermetric community felt they had struck a major blow. At Diamond Futures we don’t see the conflict as an ‘us’ vs. ‘them’ battle—that’s a petty fight that doesn’t bring us any closer to our mission: to develop the most accurate methods of determining the likelihood and level of future success of young ballplayers—both at the professional and amateur level.
It’s important that we clearly understand this mission. Our goal isn’t to sign future Major League ballplayers; it isn’t to find, establish or maintain a career with a Major League organization; and it isn’t to sign players as clients or to provide these clients with ammunition to better negotiate contracts. If it had been any of these things I am sure it would have had us tact differently along the way. No, we have but one objective—determining how to find the best way to calculate the probability that Player X will perform at various levels of success at some predetermined point in the future.
My background is in designing/developing financial modeling systems. The baseball challenge that we have described is virtually identical to those that I built a career around. It is essentially the same problem as, given a company that is currently performing at a defined point on its life-cycle, what is the probability that it will achieve a defined level of performance over a certain period of time. In both cases we have to determine/measure the current state, we have to determine the factors that may impact performance, we have to be able to assign probabilities to the potential outcomes, and we have to be able to sum the individual outcomes to determine the future value. Let’s take a look at each of these steps.
The first step is to measure the current state. At Diamond Futures we contend that as long as we can measure the current state, given enough data and an accurate method of normalizing it, we can develop definable, measurable, end-state probabilities. So realistically what does that mean? It means that if a given player has a significant sample size of data in a format that can be accurately normalized, we can determine the value of that players current performance relative to both his control population, but also to the universe of all professional ball players and value it in such a manner that is based solely on the key factors that have been demonstrated as predictors of future success. This also means that we can apply these techniques to not only Major League players and full-season Minor League players, but we can achieve similarly accurate results with players that have only performed in short-season Minor Leagues, Foreign Leagues, and even NCAA Division I. Stated in other terms, we understand the ‘real’ current value of Player X’s performance even though his only experience was in his 166AB Dominican Summer League stint last season.
One key point here, the statistical community has determined many methods of measuring baseball success: Win Shares, Runs Created and Linear Weights to name a few. We accept these measures as valid determinants of Major League performance. What these measures are not, however, are valid determinants of future performance. In the next couple of installments of ‘Do-It-Yourself’, we will detail the metrics we use.
The second step in our process is to identify the factors that may impact a given player’s ability to move from his current state to a specific future state. For many decades now, this has been the chief responsibility of the scout. While scouts do provide current state assessments in their reports, they can’t really match cold hard statistical analysis in this area. Think of it this way…if we had to measure the distance between two points, what is going to give us more accurate results: 1) The naked ‘eye-balling’ of the distance by a professional surveyor or 2) or an ordinary person armed with a tape measure? The scouting assessment of current performance relies on a lot of ‘eye-balling’…trained and experienced ‘eye-balling’, but ‘eye-balling’ none the less. But don’t mistake this for dismissing the value of scouting. Scouts provide significant value, shall I say currently irreplaceable value, in identifying things that are less easily measured. Things like: how much late life is on a pitcher’s fastball, does the pitcher throw on a downward plane, how much projection is his frame going to allow and how repeatable is his arm-slot/delivery. The scouts use their experience to gather this information, to assess the value of this information, and finally to assimilate all of the information into a prediction of what it all means to a player’s future performance. There is currently no way to gather this information without the scout.
Let’s digress for a moment and revisit the issue of scouts vs. statisticians. It would be fool-hardy for any scout to ignore the fact that systemizing data points that they gather and analyzing them for correlating factors to performance would not improve the effectiveness of scouting. If a major league team could completely standardize their scouting metrics and reporting, have a large enough staff to evaluate a significant population of players each year and collect this data for a long enough period of time to provide a base of data that could be analyzed for correlations, they could developing a modeling system that would provide quantum leaps over where the industry is today. By the same token, the statistician would be fool-hardy to ignore the fact that, as we sit here today in 2009, scouts provide irreplaceable inputs to the process. That doesn’t mean that sometime in the future that we won’t have available to us some sort of tool similar to ‘Google Earth’ that will allow some computer programmer to zoom into any game going on anywhere in the country and use tools that accurately measure the speed of the pitch, the plane of the pitch, the amount of movement at the end of the pitch and to quantify the repeatability of a pitcher’s arm-slot/delivery. Only at that point will the relevance of the scout truly be able to be questioned. In the mean-time, the astute organization is finding ways to meld both scouting and statistics—not divisively make a stand in one camp or the other.
So where does that leave us at Diamond Futures? While the scouting community is become increasingly standardized as each year passes, scouts still have different perspectives and different levels of experience. Two scouts evaluating the same player often times still come to very different conclusions. Additionally, many scouts/organizations hold their information very closely, so it isn’t readably available, yet alone in electronic format that can be easily analyzed. Finally, even if we could get past the first two issues, we don’t have a deep enough historical database of these reports to analyze and isolate positively correlating factors. So, for now, the best that we can do is try to find available, measurable, data points that might be used as ‘substitutes’ for the factors that scouting reports may have been able to provide. We have been able to identify some available ‘substitutes’ that do provide positively correlating results: for instance we can use height, weight, and body mass distribution data to ‘substitute’ for the frame projection and we can use signing bonus and draft position information to ‘substitute’ for the general scouting value of a player. Essentially we recognize that the inputs that could be provided by scouts do identify factors that may impact future performance and we have gone about finding measurable, substitute, data that can be used in its place.
The third step in the process is to assign probabilities to the desired levels of future performance. At Diamond Futures, we accomplish this forward looking technique by essentially looking backward—comparing the player with similar players from the past. We have a database of every minor and major league player’s performance for the last 40+ years and nearly 15 years of foreign league data and more than a decade’s worth of college results. What this allows us to do is search through over 12,000 individual players, and identify the approximately 1% - 2% of players that were most similar to the player in question and determine what happened to those player as time went on. We then determine the percentage of those players that performed at each level of performance we are trying to measure.
Finally, knowing the desired levels of future performance that we are attempting to measure, and the probabilities of reaching them, we can truly go about building a tree of potential outcomes and coming up with a single measure of value. By approaching valuation from this perspective, we can dispense with the whole argument of ‘ceiling’ vs. ‘certainty’, as we have a single measurement that considers both. As an example, we have players A and B. Player A has a 10% chance of performing at a level of Value 8.0, a 60% chance of performing at a Value of 5.0 and a 30% chance of performing at a Value of 3.0. Player B on the other hand has a 30% chance of performing of performing at a level of Value 8.0, a 30% chance of performing at a Level of Value of 5.0 and a 40% chance of performing at a Level of 3.0. Using our methods, Player A is a .10*8.0 + .60*5.0 + .30*3.0 = 4.7 Value player. Player B on the other hand is a .30*8.0 + .30*5.0 + .40*3.0 = 5.1 Value player. Now we do complicate things by also utilizing Marginal Replacement Level Values and the like, but essentially this is the process that we perform for every player.
The way that we will present this information on the site is that we will start out by providing our Performance Evaluations for each league as they draw to a close. The first of these will be the NCAA Division I Performance Evaluations that will come out next weekend. These are our current state evaluations that we discussed in Step 1. Then as we complete the rest of the process, we will provide our team rankings over the winter and our overall rankings and write-ups that will come out in March. Over the coming weeks we will take more in-depth looks at our techniques used at the various stages of the process. Next week we will begin this with a closer look at the factors that are used in developing our Performance Evaluations.
Posts
No comments:
Post a Comment