Business Journal

10,000 Managers Can't Be Wrong

Dr. Frank Schmidt, one of the world’s foremost meta-analysts, explains how mega-numbers can give businesses surprisingly personal insights

Businesspeople are overloaded with information about information overload. They don't need to read one more article about how many articles they need to read. And not only is there a lot of information to synthesize, but serious research results can be dry, scholarly, and difficult to digest.

QUOTE

The problem is, if executives miss the results of crucial organizational research, they lose the chance to learn about highly profitable management approaches. But most managers and executives attained their positions because they're businesspeople, not statisticians. They require an assist from meta-analysts.

Meta-analysis is the science of studying the results of many studies at once. Meta-analysts pry the meaning from vast piles of statistics, providing executives with actionable information, not just a swarm of numbers. This research is the backbone of the metrics of productivity -- the methods that provide real results.

Dr. Frank Schmidt is one of the world's foremost meta-analysts. He is also a Gallup Senior Scientist -- one of a group of leading scientists who lend their expertise to The Gallup Organization's work. Dr. Schmidt is the author of hundreds of articles on meta-analysis, methodology, selection, individual differences, and a vast spectrum of other issues in industrial psychology. He is the winner of dozens of eminent awards, including, with Dr. Jim Harter, Gallup Chief Scientist -- Workplace Management and Dr. Ted Hayes, Gallup SRI Research Director, the 2002 Best Paper Award from the Organizational Behavior Division of the Academy of Management. He has taught in universities in several countries and has been an advisor to businesses, humanitarian groups, the military, and the criminal justice system.

How can Dr. Schmidt's vast and heady academic expertise help companies improve profit margins? Well, his research gives executives valuable insights -- such as how much to value a highly experienced job candidate and how much, literally, good employees are worth. In the following interview, Dr. Schmidt shows how the thousands of studies he's studied boil down to keys every executive can use.

GMJ: What does meta-analysis offer that individual studies don't?

Dr. Schmidt: The ability to make sense of large numbers of studies on the same subject, which are typically conflicting. You don't get the same results in every study, and meta-analysis makes it unnecessary for you to even try to read all the individual studies; the meaning of the studies as a whole is condensed into the meta-analysis results.

And those results typically show that there isn't nearly as much conflict and disagreement as it appeared when you looked at each study individually. You typically find that there's a clearer meaning than you thought to the research that's been done. That's very, very helpful -- you can see what it means, and you can apply it.

GMJ: And it takes a lot less time.

Schmidt: It takes less time than reading all the studies. But even if you did read all of them, you still couldn't disentangle them; you still could not interpret them.

I was thinking the other day that you can compare meta-analysis to a search engine. We're all swamped with information overload today. There's an incredible amount of information on practically everything. Search engines allow you to find particular pieces of information instantly that you might otherwise never find in a year of searching through the library.

Meta-analysis tells you what very large amounts of information on a particular subject mean. It eliminates the variation from sampling error and other statistical artifacts in studies and gives you the bottom-line results of a whole lot of research. Search engines and meta-analysis are both ways of managing the incredible information overload. The human mind simply cannot deal with that much information, so we need tools to make it possible.

GMJ: So meta-analysis allows you to study the results of surveys with 10,000 people and find the two or three real problems that hamper employee engagement?

Schmidt: Well, take the Gallup Q12 employee engagement survey for example. It's often administered to thousands of people at a time. But the Q12 only has 12 items, and each one of those items is very specific -- all the questions have to be actionable if you want results. If a business unit has a low score on one of these, then it is clear enough what the problem is so that you can take action, no matter how many people take the Q12 survey.

Next, the overall company results from the survey should not be distributed to the company as a whole. Instead, the feedback should be broken down by business unit. Remember, if you're going to give feedback to an individual manager, it has to apply to that manager's unit. The company scores as a whole should be looked at by higher level executives, and the scores could also be compared to other companies -- to norm groups -- to help benchmark results. But the real value is at the level of the individual manager and his or her department. Distributing results at the workgroup level ensures that a manager receives feedback that provides guidelines for what should be worked on, what should be improved.

GMJ: There's often some skepticism in the ranks when these surveys are administered. But by the second or third time, I've seen pure managerial evangelism for engagement studies.

Schmidt: Well, you know, I think the Gallup Q12 has a generic effect in improving managerial performance. I think it stimulates an overall improvement in the quality of management. And if there is some problem that doesn't exactly fit into any of the particular questions on the Q12, that problem will probably be addressed by the manager as the manager becomes better in general.

GMJ: Do you have to give the surveys over and over again to measure change, or can you do it once and call it good?

Schmidt: These surveys are rarely one-shot applications, and they really never should be. Company surveys -- the Q12 or any other -- should be done periodically; they're typically done every year in many companies. That way, managers have the chance to see an improvement in the engagement scores -- item-by-item and overall, or question-by-question and overall -- that tells them the process is working. That's because these survey questions not only have to be actionable, managers must review the results periodically to determine how well the actions they took worked.

GMJ: So a meta-analysis can show you two serious problems for productivity. But what's the relationship between workforce productivity and good employee selection?

Schmidt: Well, that's a question related to practical utility. Many, many studies have been done on that, and they show that good selection greatly increases the average level of output of employees on the job. And that's the bottom-line nature of the utility metric. Going from a poor selection procedure to a good selection procedure can easily increase the average output per person on the job by 10% or maybe even 15%. To some people, that doesn't sound like a lot, but it translates into very large numbers.

GMJ: One of the more traditional methods of employee selection is to confirm that the candidate has lots of experience, then do interviews to see what the candidate is like. What are your thoughts on that?

Schmidt: Well, it's not the best method. You mentioned relevant experience. We have done a lot of research on that. Job experience does have validity for predicting job performance, but that validity fades out with time. After five years on the job, it will be the more intelligent people with better personalities who will be the better workers -- not those who initially had more job experience.

QUOTE

GMJ: So if you have to make a choice between a conscientious, intelligent person with three years of experience and a real jerk who has ten years of experience, you're better off with the first person?

Schmidt: Under those circumstances, I would prefer the one with three versus ten years of experience. The validity of experience for particular job performance fades over time, but the personal characteristics validity does not fade. Initial learning on the job is pretty steep during the first five years. It makes a big difference whether you've been on the job one year or three years.

Learning through experience, however, levels off with time -- and for many jobs, that's at about five years. So a person with ten years of experience will not, on average, have better job performance than a person with five years. It might be a longer period for more complicated jobs, but for the typical mid-level job, that's about the breaking point -- people have learned about as much as they are going to learn about how to do that job after the fifth year. The difference you can attribute to experience will fade away and will no longer affect performance. What will become important will be mental ability, personality, and conscientiousness -- personality traits. These traits do not fade away. That is, their predictive ability continues.

GMJ: You've done a lot of research into "individual differences." What's that?

Schmidt: Maybe I should start by telling you what the alternative is. There are a lot of people in psychology who look for general laws -- the ways all people are alike. That's the opposite of the individual-differences approach. You're looking at differences in general intelligence, differences in interest, differences in values -- these differences are just gigantic. It's important if you're looking at the ways people respond, for example, to different training techniques. And it really does call into question whether it is possible to have a general law that applies to everyone. I said this in class the other day, and my students were shocked -- the range of ability in the top 1% of a group is greater than the range of ability in the middle 98%.

GMJ: How can that be?

Schmidt: Because it's the shape of the normal bell curve. The normal bell curve just keeps going further and further out. You know, in the top 1%, a person who is right at the cutoff can be different by four standard deviations from someone who is further out on the curve. There aren't very many people way out there, but there are some, and they are important.

GMJ: How can knowing the individual-difference bell curve help businesses?

Schmidt: For example, we were studying the practical value of selection procedures, and we found that supervisors were much more concerned about low performance than high performance in their employees. Their attention tended to be focused on the people who were a problem. They thought it was more important to increase the performance of the people at the bottom, just bring the low end up by a certain amount. As a result, they didn't appreciate their high performers as much as they should have, and really great performers didn't get much attention because they didn't cause problems.

GMJ: That's just disastrous to an organization -- if managers spend all their time and energy on people who barely produce, they won't find time to make their stars into superstars. What can change that focus on the low end?

Schmidt: High levels of competition in the industry. It's a shame that it has to get to that point, but if some other organization is breathing down your neck, you start focusing on the high performers because these are the people who can meet the challenge. As I mentioned, just a 10% improvement in productivity translates into big numbers, so paying attention to the top of the curve can be tremendously beneficial.

GMJ: As long as you don't hire jerks.

Schmidt: Well, we don't use terms like that. But yeah, the key is to hire the right people.

-- Interviewed by Jennifer Robison

Gallup http://www.gallup.com/businessjournal/10882/10000-Managers-Cant-Wrong.aspx Gallup World Headquarters, 901 F Street, Washington, D.C., 20001, U.S.A +1 202.715.3030