The Orioles Warehouse

The profiler

Host Tom Davis then turned the conversation to the Orioles' use of psychological tests to assess players. Ritterpusch explained that the team has a database of test results for draftees going back 30 years to the time when he was scouting director in the '70s. He said that during the time he was away from baseball, he remained in contact with the organizations that administered the tests, and through those channels he was able to procure test results for various players, including Roger Clemens, Doc Gooden, and Rafael Palmeiro. When he returned to baseball, he found that the database of test results had grown large enough to identify useful patterns—specifically, which personality traits were common to the most successful players. In his words:

So I think what changed is we developed a larger body of knowledge ourselves, and then when I came back I was able to build on that, particularly last spring, and look at large selections of players over time that were successful and unsuccessful. We began to become more refined, smarter. We were able to increase from three or four predictive things to about a dozen predictive values.

The last part about obtaining "about a dozen predictive values" makes sense in that the psychological test used by the Orioles is designed to measure eleven traits. According to Eisenberg's article:

The test the Orioles use is the Athletic Success Profile, overseen by the Winslow Research Institute of Antioch, Calif. It asks 110 questions intended to measure 11 attributes: drive, aggressiveness, endurance, leadership, self-confidence, emotional control, mental toughness, coachability, conscientiousness, responsibility and trust.

The Athletic Success Profile (ASP) and an earlier version called the Athletic Motivation Inventory have been administered to potential draftees by the Major League Scouting Bureau since 1974. The Winslow Research Institute web site does not have information about the ASP, but it has a short description of a more general-purpose test called the Winslow Success Profile. The WSP test overview lists the same eleven attributes as the ASP plus three composite personality characteristics: competitiveness, self-control, and dedication. It is possible that the Orioles use some of these composite traits, or that they synthesize other traits using their own analysis of the test.

Making the grade

Asked to expound on the player rating system that the team has developed, Ritterpusch was candid and descriptive—to a point.

Well, in a broad sense, as far as our orientation, [our rating system] is to find talent and to identify, categorize the talent in regard to a major-league inventory, major-league standards. So in the broadest sense, we identify players who we feel have the talent, both physical and psychological, that would make them stars, which for us is the top ten percent of the game, then first-division regulars, which is the next twenty percent, and so on down, second-division regulars, who are the next thirty percent.

Here Ritterpusch yielded some insight into the team's scouting process. His high-level overview of the Orioles' player-evaluation system indicates a tools-based style of grading physical talent that is combined with psychological data to produce a more refined projection of the player's long-term potential. He did not elaborate on the evaluation of players' physical abilities, so perhaps he is not as intimately involved in that aspect of the scouting operation. But Ritterpusch's level of knowledge about the process shows that he works closely with the club's scouting department and is not just a test expert and number cruncher. Still, number-crunching is evidently one of Ritterpusch's strengths, and the psychological rating system he described has an empirical basis and an intuitive, quantitative scale. Again, from the interview:

We know that these are people with exceptional physical ability, but when they are competing with each other, then certain psychological traits make all the difference in the world. And we do grade, we do lump them, although we initially get the grades and establish a "key trait coefficient," a coefficient using the traits that matter the most in baseball, and those are graded then on a percentile. So it's a one to 99th percentile against the body of professional baseball players historically that we have tests on. To simplify it for ourselves, we then break it out into one to five, five being high, and then "five-plusses," so that just in terms of traits, you'd have a [Josh] Beckett and [Roger] Clemens have five-plus trait packages.

Since the weights assigned to individual traits remain a mystery, the "key trait coefficient" of which Ritterpusch spoke represents the Orioles' secret sauce in the psycho-measurement arena. As Ritterpusch explained, the coefficients are then translated to a linear percentile scale and finally are divided into quintiles. The ensuing one-to-five shorthand classification is analogous to an academic letter-grading system, with a rating of one corresponding to an E, five to an A, and five-plus equivalent to an A-plus.

To further validate the testing approach, Ritterpusch pointed out that the Orioles re-administered the test to Palmeiro this spring and discovered that the results were identical to his pre-draft results from 1985. The agreement of the results corroborated the widely accepted assertion that personality traits are stable and enduring from the teens into adulthood. Given the colossal amount of uncertainty involved in predicting the fortunes of young players, the long-term reliability of the personality trait profile makes it a significant piece of information to consider in the scouting process.

Diving for pearls in Dave's hard drive

According to Eisenberg, Ritterpusch has taken nearly 10,000 players' worth of psychological test data and apparently has done some form of data mining to identify the traits of highly successful ballplayers.

Knowledge discovery through data mining is a tool that was not practical thirty years ago because of the limitations of computer storage capacity and processing speed. Modern hardware and software, however, have made it possible to perform analyses of large data sets in a matter of seconds. The only limiting factors to obtaining useful results are the amount and quality of the data, as well as the validity of the statistical model used in the process. The Department of Defense's aborted Terrorism Information Awareness project (which originally was called "Total Information Awareness") is perhaps the most widely known application of data mining, and Ritterpusch, with his experience in the defense establishment, surely is aware of it and related endeavors. Big businesses around the world have been adopting data mining techniques in recent years to identify trends and patterns that would otherwise go unnoticed by the human eye. For the Orioles, who have an immense amount of capital invested in a relatively small number of employees, it is simply good business sense to make use of such technology.

Psychological testing of athletes has also gained acceptance over the last several decades. It was in its infancy during Ritterpusch's first tenure with the Orioles—the Athletic Motivation Inventory, which was the initial version of the ASP, was developed in 1968—and in the early 1970s the set of test results had not yet reached the critical mass necessary to produce statistically useful patterns. Moreover, the psychological data only became powerful with the knowledge of how the players performed at the minor-league and major-league level. Over thirty years' worth of performance data on athletes has enabled Ritterpusch to make concrete correlations between psychological attributes and on-field performance. Ritterpusch did not explicitly mention statistical methods like ANOVA and multiple regression analysis, but one can infer that something along those lines was probably used to refine the Orioles' grading system.

Ritterpusch and the Orioles have exploited a rich set of test data to gauge the significance of certain personality traits as predictors of baseball success. They have done it using a selective sampling approach: first choose a subset of position players or pitchers who have attained a high (or low) performance level, then identify psychological attributes common to those players and compare the results to the greater population to identify the "key" traits.

Baseball's elite minds—on the field

Ritterpusch was quick to name several players who scored highly on the Orioles' proprietary scale. In addition to Beckett and Clemens, he also cited many other current and former stars who rated well psychologically, including Palmeiro, Todd Helton, Derek Jeter, Scott Rolen, Matt Williams, Barry Larkin, Bobby Grich, and Fred Lynn. Asked about Jeter in particular, Ritterpusch commented that Jeter "has one of the classic champion profiles.... In the key traits he's a five-plus." The Sun article also refers to Mike Mussina and Barry Zito as having exemplary profiles.

Since Ritterpusch returned to the fold, the Orioles have loaded up on prospects who rate five or five-plus on their scale. Most are pitchers: Adam Loewen, Kurt Ainsworth, Ryan Hannaman, Don Levinski, and Chris Ray are among them. Daniel Cabrera graded well on a Spanish version of the test that the Orioles recently developed, and his profile convinced the Orioles that he could mentally handle the callup to the majors from Double-A, according to Ritterpusch. 2003 first-rounder Nick Markakis (who pitched in college), 2002 draftee John Maine, and left-hander John Parrish also rate as fives on makeup.

Team psychic?

One might be led to wonder why the Orioles have been able to acquire so many psychologically optimal pitchers but so few hitters. According to the Sun, Ritterpusch found that "distinct patterns [of traits] exist for starting pitchers, closers and position players." Ritterpusch did not elaborate much on these differences in his TV interview, but he did say that it would not be "practicable" to field an entire team of fives and five-plusses. The reason for this is the lack of uniform distribution of talent across positions. That is, at certain positions there are not enough candidates with both an excellent psychological profile and the physical ability to play that position. However, Ritterpusch was optimistic that the Orioles could eventually have an entire pitching staff of fives and five-plusses because "the pitching function is interchangeable."

While Ritterpusch referred to "the overwhelming majority" of star players having high scores in "the same collection of key traits," he neglected to address the converse of that line of reasoning: of all the prospects who possess high scores in the key traits, how many become stars? Not nearly as many, probably. Although the distinction is a subtle one, it is meaningful. One of the oft-repeated caveats uttered my introductory psychology instructor was that "correlation is not causation." Thus, the fact that the same traits show up in the profiles of most stars does not necessarily mean that players who score highly in those traits will inevitably become stars.

Ritterpusch, to his credit, is aware that psycho-profiling is not the end-all of prospect evaluation. "Good scouting to cover the physical ability is paramount," he said. And he became especially opinionated on the importance of drafting for talent instead of need. In the television interview, he put it this way:

With a young player, whether it's the draft or a trade, you do not want to [go] for a need at the lower levels because you will screw up your evaluation.... What you do want to do is get talent. In any year there's not that much talent. You want to get that talent. And then as players rise, and as you go to acquire free agents or make trades, then you go for specific needs with where you know the commodity. But you've got to have a body of talent to work from.

In the Sun article, he said, "The test evaluates a prospect's chances of maximizing his physical potential; that's all." So the accurate evaluation of other aspects of a prospect is still tremendously important.

Reading between the lines

Also notable is what Ritterpusch did not mention in Sunday's interview. Predictably, he was not eager to disclose the specific personality traits that the Orioles consider important. When prodded by Davis, Ritterpusch conceded that the ability to handle adversity (which falls under mental toughness) was "obviously one of the critical traits, particularly to position players." Reading his comments on Flanagan and Murray, one can surmise that drive and emotional control are also important attributes. But in the Eisenberg article, Ritterpusch pointedly asserted that drive is not as important as most people think. "The old cliche about 'the guy who wants it the most will get it'—it's a myth," he said.

Adherents of the sabermetric approach, which was popularized by the writer Bill James and exploited by the Oakland A's, may be somewhat dismayed that Ritterpusch did not discuss the statistical evaluation of players' on-field performance. This seemed odd, given that his title is director of baseball information systems. Was this because Davis never steered the conversation in that direction, instead choosing to focus on psychological profiling? Or is it that the Orioles have not developed an effective system for using performance-based statistics in evaluating prospects?

Next: More analysis of the Orioles' methods of player evaluation.