Assessing the Stability of Your CliftonStrengths Results

Jim Asplund, Chief Scientist, Strengths-Based Development and Performance Impact Consulting, was our guest on a recent Called to Coach. Jim discussed the newly released paper he authored, "The Stability of CliftonStrengths Results Over Time" (PDF download), in which Gallup examined the test-retest reliability of the CliftonStrengths assessment, and found it to be remarkably stable. Key points in the discussion included:

• the circumstances under which it may be helpful to retake the assessment
• what people mean when they say they "don't like" their CliftonStrengths results
• how the new research validates previous Gallup research on this topic

Below is a full transcript of the conversation, including time stamps. Full audio and video are posted above.

If somebody asks you, "Do I need to take it again?" Most of the time, no. But what we can now say is, "But in these situations, it may make sense for you."

Austin Suellentrop, 21:28

I've met people who look at this [their CliftonStrengths results], and say, "That's just not me," and they could kind of make a case for that. Now, then, you kind of wonder, how did that happen? ... sometimes it's just a failure to acknowledge something about themselves that maybe they don't like.

Jim Asplund, 39:23

I'm really excited that when we publish this, this latest research, we can still stand with confidence and say, "The assessment is stable over time." In fact, in our 6-month study, the reliability has actually gone up slightly.

Austin Suellentrop, 21:11

Jim Collison 0:00

I am Jim Collison, and live from our virtual studios around the world, this is Gallup's Called to Coach, recorded on June 17, 2020.

Jim Collison 0:20

Called to Coach is a resource for those who want to help others discover and use their strengths. We have Gallup experts and independent strengths coaches share tactics, insights and strategies to help coaches maximize the talent of individuals, teams and organizations around the world. If you're listening live, we'd love to have you join us in our chat room. It's actually -- there's a link right above me there on the live page, take you to our YouTube instance. There's a chat room there. Log in and ask your questions live during the program. If you have questions after the fact -- many of you are using this email address now; thank you for doing that -- you can send us an email: coaching@gallup.com. Don't forget, if you're on YouTube, you can subscribe to the channel right down in the corner there. Hit the notification bell that'll just give you a notification whenever we publish anything new. And if you want to listen to this is a podcast, search "Gallup Webcasts" on any podcast platform. Austin Suellentrop is our host today. Austin's the CliftonStrengths Portfolio Manager here at Gallup. And Austin, it's always a great day when I get to podcast with you on Called to Coach. Welcome back!

Austin Suellentrop 1:14

Thank you so much! Glad to be here. It's always fun.

Jim Collison 1:17

Let's take a second and introduce our guest today.

Austin Suellentrop 1:19

Yeah, I'm really excited. When, when I first started working in this role a little over a year ago, one of the biggest pleasures I had was getting to know and getting to work more hand-in-hand, with with Jim Asplund. Jim, among a lot of things, he plays a lot of roles at Gallup, but one of the things he spends a majority of his time working on is leading the research efforts around CliftonStrengths. So if you think about all the things we're doing constantly in researching the impact of strengths, how we -- how it impacts organizations and communities and countries and individuals, Jim is sort of at the charge of that. So Jim Asplund, thank you so much for joining us today.

Jim Asplund 1:56

Thank you, Austin, that was very nice.

Austin Suellentrop 1:57

Well, well, I've got -- I've got to be nice to the public, Jim, right. I mean. So Jim, Jim, Jim is going to be joining us here today to help talk through -- as Jim Collison said, and it's like Jim and Jim -- as Jim Collison, said a really exciting topic. And that's the idea -- probably the most, one of the most common questions we are asked at Gallup is, I took my assessment a while ago. I took the assessment, I don't know, a couple years ago, should I take it again? Or I took the assessment when I was working in a different company. Should I take it again? And so that question is one of those most frequent things asked, and it's usually sort of accompanied by a curiosity of Will my results change? How consistent will these be over time? If I wait 2 years, will my, will my strengths be different? Right?

Austin Suellentrop 2:50

And so that's a really big question that we've been asked for many, many years. And luckily for us, we've got partners like Jim Asplund, who are curious about the same things and have been looking into them. So we've got a new research paper that was just published this past week. It's available -- I know Jim Collison's put it in the chat -- a link to it in the chat room on gallup.com/cliftonstrengths. If you go under Resources, [then] Guides and Reports, there you go: the latest stability research of our assessment over time ["The Stability of CliftonStrengths Results Over Time"]. So today, we're going to unpack that a little bit. We're going to talk about some of the big things that stood out in the research that Jim led, and hopefully answer some key questions you may have about how it impacts your conversations with clients and your coaching. So Jim, first and foremost, what led us -- why did we do this updated research?

Jim Asplund 3:40

Well, it's a fairly responsible thing to do when you've had, say, 20-some million people take it since the last time you updated the research. You know, it's, it's a luxury we have that we have a lot of data and we should make use of it. And the other thing that happened, just longer time time has elapsed. So, you know, and by that I mean, we had longer gaps between the first and second, or last, time someone took the assessment. So we could look at retest or stability over longer time periods than we could years ago because it's been around longer now.

Austin Suellentrop 4:15

Yeah, I think it's, it's a big point to highlight. We first published the test-retest reliability in the technical report -- what, 2008 or so was sort of the last the last time we refreshed this? So we've -- it's been 10-plus years. And we've gone to now, at the time you conduct this research, we had a little over 21 million people in our database, right? So millions more people completed it. And I think for all of us to understand, a lot of the people in that database may have taken it the first time as a college student. We have an emerging -- a big strong presence on college campuses. And that they may have been you know, in that 18- to 22-year-old age bracket in college. It's been 5, 7, 10 years for some of them now. It allows us to study the change there. So good call us there.

Austin Suellentrop 5:07

I think, to your point, we do have so much more data now, and our database is growing so much we can learn from that. And it is responsible thing for us to do, so --

Jim Asplund 5:17

A point I forgot to mention is we had a lot more data outside North America. So we could look at other populations too. So that was --

Austin Suellentrop 5:26

Yeah. Wonderful. Wonderful. So what when we look at this, right, we've got a more global presence. We've got longer time periods. We've got more data points to reference. What was the -- what did we learn? What, what's new in this report that people are gonna want to pay attention to?

Jim Asplund 5:43

Well, I think the, you know, the main thing that's new is that we just, we have a lot more things we can explore. So we could look more into stability over longer periods of time, stability for people who took it in 2 different languages, stability among subpopulations, by language, gender, race, that sort of thing, country. That's kind of what's new. What's not new is that the assessments' results are still very stable over longer periods of time. In fact, I think over the same period of time, the reliability is actually a little higher than it was when we when we published it 2008. And so, I mean, nominally, it's not a lot. But, but it still looks really good. But now we can kind of dig into more, you know, some of those hypotheses where possibly something had happened that would make the scores change more than for the average person. And, and, but for the vast majority of people, they didn't change much.

Austin Suellentrop 6:43

Yeah. So what I, what I've heard from you both there and in our conversations sort of leading up to this and as we've worked through the research, if somebody was referencing a certain number in the past of how stable is the assessment over time, right. There's a number they could reference in the past. If somebody asked that question now and said, Hey, all right, Jim, what's the stability ratio of the CliftonStrengths assessment? How would you answer that?

Jim Asplund 7:11

Well, the published result in 2008 for the overall profile, the test-retest correlation was .7. And now it's .73. So like I said, slightly higher over the same time period. And it's a, that's a sort of a 6-month time period, which is pretty common in the literature. It attenuates a little bit over time. So if you go out longer periods of time, it drops a little bit, and the paper goes into that in detail. But even over, you know, 7 to 10 years or more, it doesn't drop way below that. So, but like I said, it's a little lower. But so that .73, I think, is a kind of the number that's earliest in the, in the paper. It is kind of if you want to pick a number to kind of encapsulate what the thing says, that's the one I'd use.

Austin Suellentrop 7:58

That's the one, OK. So for everybody listening, page 3, Table 1 of the report, there's your, there's the sort of the updated, refreshed version of the same research we did in 2008. So if you looked at that, so that sort of six-month reliability of the whole, of the whole profile, that's listed there, page 3, Table 1, and it's .73. So I know a lot of our, of our coaches, a lot of our clients, they want that number. They want that number because they're trying to make the case for strengths, or they're trying to answer questions from stakeholders and be able to say Why, you know, how consistent is this?

Austin Suellentrop 8:31

And what's great is in the past, we had one number to provide, right? We had that one research point, we had that sort of one study. This new research gives us so much more nuance, right? It gives us so much more depth and variety of angles we can look at this. So in the past, I know, I think one of the, one of the common things we would say if people asked us, Should I take it again? Our lead answer for many, many years was "No, you don't really need to. It's not going to change much." Right? It's not gonna, your results aren't going to significantly or meaningfully change over time. So there's not a need to, to take the assessment again. Now, Jim, how would you answer that question of, Should I take the assessment again?

Jim Asplund 9:12

I think you still -- the sort of default position is you don't need to. We have explored some circumstances where maybe it's worth some thought. But that's a distinct minority of the population. And we kind of enumerate those in the paper, but some things you might -- some of it's, you know, some of the -- so in the study, we looked at everybody who took it more than once. We didn't really care why; we didn't know why, for the most cases, why they did it again. Some people took it in a, in a different language the second time. Now I don't -- anecdotally, my experience has been people who take it the first time in a different language, they take it in English when English is not their first language. And that's pretty natural. If you're a college student, for example, a lot of those are students studying in North America who are doing everything else in English; that's just a natural thing for them to do. And, and then they take it years later or months later in, in their native language, and the results are a little different. And, and that's to be expected.

Jim Asplund 10:16

It's actually -- the difference isn't big, but there's something there. Other people took it again because they couldn't find their results. So, or they didn't want to do the work to find it. It was easier just to take it again. And then other people, you know, maybe weren't taking it that seriously the first time. I think we've all met those people. I certainly have. They didn't necessarily know why they were doing it. Somebody gave them a code oftentimes, and they were in a hurry or they were distracted or whatever. And they took it before. If you think that was the case for you, you know, yeah, paying attention probably helps. So maybe that's a good reason to actually do it again. And then you shouldn't be surprised the results are different because you're investing more in your responses.

Jim Asplund 11:08

And so we explore some of that; it's hard to know, right? So you can look at things in the data. You know, how long did they take to answer the questions? How much did their answers vary? How did their sort of response style change from, you know, the first one to the second one? And you can infer some of the reasons for that. I mean, obviously, we usually don't know. But I think that, that's the most common circumstance is they really, you know, they weren't paying attention or taking it seriously.

Austin Suellentrop 11:36

And Jim, I think that's a great distinction. So there are, I think, there are some times now that we would say, Hey, if this is, if this is your situation, you may want to consider it. It's still not a blanket "Yes, you should retake it." But as coaches, it's a, it's an invitation to have a discussion with our, with our client, right? That if they're saying, Hey, would my results change or should I take it again? It's opening the door for us to discuss with him what's driving that? And so in the paper, you outline some some great questions to ask. Right?

Austin Suellentrop 12:11

So I just want to cover those 3 questions you outlined to ask if somebody prompts that question. The first one is, "How old were you when the first -- the last time you took the assessment?" Right, because, as you mentioned, results sort of correlate a little bit, we know that they will change more likely, like, from the time you're younger. Once you hit a certain age, it doesn't change a whole -- as much or as significantly. So if they were 18 to 22 to 24 the first time they took it, and they're in their mid-30s now, maybe it would make sense for them to retake it. Right. The second question was, "How long has it been since you took it last?" So from that point, if it's only been 6months, pretty much regardless of situation, that wouldn't make a whole lot of case for us to recommend retaking it. But if it's been, what, 10 years, is that a number?

Jim Asplund 12:59

Yeah. And they interact, by the way. So if you were 50 when you took it the first time, 10 years might not change that much. If you were 19, and then 10 years have elapsed, that's different. So there's a developmental piece to that. And the literature and, you know, kind of personality and traits, very similar to this. They just get more stable as you get older. And, and we see that with strengths results, too, just like you would with the Big 5 or something like that. And, and so but, you know, so I would I think of those kind of as a pair: you know, How old were you and how long has it been? So, so at my age, you know, you know, don't bother, unless you got a real compelling reason. But, you know, if you're, if you're a kid the first time, and it's been a few years, it might be worth looking at.

Austin Suellentrop 13:51

Great. Wonderful. And the third question you outline is that idea of focus; "How much effort and attention did you give it the first time?" So you spoke to some of the potential distractions that could be causing that. I know one of the common questions I'm asked, Jim, is if I'm in a state of high stress, or I'm in a state of, you know, of anguish, or I'm really overloaded, how could that impact -- would that be the kind of thing that would play into that distraction mode you're talking about?

Jim Asplund 14:21

Yeah, sure. And, you know, and all we have to go on is your answers to the questions right? And so what can alter how you do that? And if you're distracted, if it's you're taking it in a noisy environment, if you were worrying about something else, if you were under a lot of stress for some reason, that could alter your responses to the items. And, and now a lot of times people won't remember that, and so I, you know, it's, it's kind of a thing you almost run across more when people wonder why their results changed. You're like, well, you know, were you, were you having problems then? Or did you take -- I mean, cause, you know, I've got a 17-year-old still in my house, and they're multitasking all the time. And if, if he takes it in college, I'm not 100% sure he wasn't also playing Fortnite and texting with 8 people at the same time.

Jim Asplund 15:19

And so you know, that's real life, you know, that happens. And they may not even remember that, but it is something to explore if people are wondering, you know, are these right for me still? Or should I take it again? I think that the first place to go is just measurement error from things like that.

Jim Collison 15:38

Right. Jim, can you get let's talk about the opposite side of things. We often talk about being distracted or being young or life circumstances. But what about those who have had intense strengths coaching or training, who understand the language of strengths, who may even understand themselves better. Does that, does -- when we add the element of coaching or training to someone, does that, can that affect the way they answer the questions and maybe add some bias in?

Jim Asplund 16:09

A little bit. And I'll tell you why I think that. Most of our items, fact, I think, almost all of them, you can look at them and not know what theme they're measuring, even if you had the time to sit there and stare at it. They're designed that way. There are a couple, and we see that in the data a tiny bit, where heavy exposure to our content -- and that's Maximizer and Individualization -- may have changed people's retest a little bit. And I think that's just the nature of people being really interested in it. They can spot that now when we're asking about it, which, the first time they, if they were not previously exposed to strengths content, that would not have happened.

Jim Asplund 16:55

We're, you know, we're looking into that. I don't think there's a lot you can do about some of it. Some of it I think we can, now that we know. But I say, again, that those effects are small. And in fact, if you want to check it out, it's kind of towards the end of the report. And it's a useful point to bring up because there's kind of two ways to look at stability. Right. So there's Did the scores to the questions change by theme, you know, so did they change by theme? And then, the way we report scores is a ranked ordering of these things. Right. And the interesting thing is, if, you know, if you ever played Jenga, you know, you pull out one thing, everybody moves, right.

Jim Asplund 17:33

So if your scores change on one theme just a little bit, it moves everybody. And so it makes it look, in a way, the way we give out the results makes it look even a little less stable than it really is. Excuse me. So if you look at people's responses to the themes over time, it's way more stable than it looks like when you look at the kind of rank-order thing. So there's a table in the back that kind of shows the distribution of scores. And they're all way over 90%, you know, similar -- over 95%, I think, all of them, from Time 1 to Time 2 in terms of just the raw scores. And so the thing is, is that changing one or two of those and more often than not, that's even a little strong, but the themes that were the most often the cause of that would be those two that I mentioned: Maximizer and Individualization. You'll see that they have the slightly lower similarity Time 1 to Time 2. They're still -- I'm not looking at it right now, but like 95%.

Jim Asplund 18:27

So, so, so, you know, we're not changing our responses much; it's just that the way we report these things moves the water that you're -- all the other themes are in a little bit, and so. So that's another thing to know is sometimes these themes move around a little bit and not to get too wound up over small movements, right, you know, that, you know, if it was No. 1 before and it's No. 2 now, that -- those could be almost exactly the same. And, and, you know, if it was No. 3 and now it's No. 5 or 6, still towards the top. And, and, and they may not have changed at all. May have -- something else may have moved him. So --

Jim Collison 19:02

And I think for our coaches, this is an excellent opportunity to ask some questions why. If we have two reports, they're working with someone and those, those people are asking, like, why did this move? Those are questions to ask back. "I don't know -- what, what, tell me a little bit about how you're being successful in what you're doing. The Name it, Claim it, Aim it philosophy that we kind of work with, when we only compare the themes, we stay on the Name it, Name it and Name it, which is -- like we never get off of the OK, what are we actually doing with this? And I, I think that pushback sometimes or that that questioning we get from people for coaches is a great opportunity to ask some really deep questions of, "I don't know; tell me some things about you." Because on the surface it's not going to make any sense. We really need to -- to dig in --

Jim Asplund 19:45

Actually, if you're still only looking at Top 5, I'm hoping that people who now have access to -- a lot more people have access to the whole sequence of 34 -- it's easier to spot those little movements that may have popped you in or out of the Top 5. So you can see that it didn't go away; it moved slightly down the street. And, and not to get wound up about something that's probably not real.

Austin Suellentrop 20:08

And that, Jim, I think that's a really, really key point. In my experience, having coached, you know, hundreds of people around strengths over the years, it's, it's amazing how often somebody would bring 2 Top 5 reports to me, and say, "Look, my Top 5 results the second time are so different than the first time." And then if we looked at the 34 for each of them, the Top 10 were identical. And the order had shuffled a little bit in the Top 10. And to your point, it's -- oftentimes it's not even that the theme itself has changed or that their -- the measurement of their talent has changed; it's that one or two themes shifted up, and it just bumped everybody else down.

Austin Suellentrop 20:47

And so I think that's a really important thing for us to know as coaches, that it's sometimes, what can appear to be a big shift for people is really not significant shift, but one or two things that have changed. But because of how the results are reported, it can appear that way. So I think as coaches, and to your point, how do we ask questions? How do we use this to inquire? I'm really excited that when we publish this, this latest research, right, we can still stand with confidence and say, The assessment is stable over time. In fact, in our, in our 6-month study, the the reliability has actually gone up slightly, from .7 to .73.

Austin Suellentrop 21:28

If somebody asks you, "Do I need to take it again?" Most of the time, no. But what we can now say is, "But in these situations, it may make sense for you." Right? And that's the beauty of this. It gives us more nuance and you know, you now, as coaches, have Gallup's research to back your question, to back your curiosity in saying, Look, according to the research, there are some situations that it may benefit you to take this again. And so it allows us to have more confidence, I think, in having those nuanced conversations. We understand that for a long time, our blanket answer was, No, you don't need to retake it. And that -- we taught that; that's how I was taught. That's how I started my career as a strengths coach. And that still applies most of the time. But when we do need to retake it, we have a better understanding now of some of those situations that may, that may benefit from it. Is that a fair summary, Jim?

Jim Asplund 22:22

Yeah, yeah, exactly. And, and it's, it's important to remember that, you know, there, there's the reality of what your strengths are, and then there's our measurement of it. And our measurement of it's very good. But there's a limit to how accurate you can be and not make the test take all day. Or ask things of people that maybe they don't want to tell you. And so, so, I think, you know, we need to be realistic in the sense that, you know, this is a pretty quick assessment in terms of time it takes people to take it. And it's accurate. But, but to be realistic, in the sense that it is an estimate of your -- a good estimate, but an estimate of your themes because, you know, we can't pry inside you, or go into, you know, absurd detail and be, and realistically expect people to still take it.

Jim Collison 23:16

Yeah, over that over that length of time. Jim, Justin had asked a question in the chat room, and both of you referenced it just a second ago, so I think a good time to cover this. .73 doesn't mean anything to a lot of people, like. So how do we, what is that, like if we were to put that into nonmathematical words, Austin, let me let me start -- well, no, Jim, let me start with you. Let's put this in nonmathematical terms. What's .73 really mean then?

Jim Asplund 23:43

Well, it's the correlation of your results, which, you know, everybody will go, "OK, great. Now, what's that mean?" So, if you want a real-world example, you know, height and weight are like .8, you know, for the general population. So, you know, we're in that neighborhood of, like, the taller you are, the more you weigh -- sort of pretty correlated things. The -- I mean, if you plot them, you know, on a curve, you know that line's gonna be pretty straight. Where, you know, being high on one is gonna be pretty high on the other one. The, you know, not using numbers is kind of limiting in terms of a way to talk about it. But I would say, you know, it's a lot stronger than most of the things you encounter in everyday life outside the lab. So --

Jim Collison 24:29

Austin, would you add anything to that?

Austin Suellentrop 24:31

Yeah, I would say, point -- if perfect correlation, meaning every result was exactly the same, would be 1.0. Right? So nothing changes time over time is 1.0. .73, the layman's definition -- and Jim, you can correct me if I'm, if I'm way off base with this, right; that's why we do this -- is that your results are about 73% consistent.

Jim Asplund 24:54

That works.

Austin Suellentrop 24:55

Right? So, you know if perfect consistency time over time is 1.0, at 100% consistency, .73 says it's about 73% are the same, which means the order is going to be 7 out of the top 10 are going to be about the same. Right? You may see shuffling in 2 or 3, is, is sort of the the the layman's way I've explained it over the years. Again, I think, being comfortable in knowing that there's not an assessment out there that I'm aware of that's got a perfect 1.0. Right? That -- there, that, that .8 is what is sort of like something with height and weight, which, you can imagine, has far more data points than we have, right, to be able to plot out. .73 is remarkably strong. We were very proud of the .7 that we published in 2008. So now that we're able to publish .73 is even stronger.

Jim Asplund 25:46

You know, another way to think of it, you know, you just kind of prompted a thought, which is always dangerous: If you think about, How could you be sort of perfect? That would require answering all 177 items the exact same way. And I don't know if anybody's ever tried to do 177 things in a row the exact same way, even a couple hours apart. Good luck. It's hard. Now, if we had a 10-item assessment, yeah, the odds would go way up that you could get it exactly the same the next time. So, you know, so I think, you know, people should understand the degree of difficulty of even just asking a respondent to do that on purpose would be really hard. And so, so if you think about 177 decisions, one or two differences are going to drop that from, from perfect, and that's, that's fine.

Austin Suellentrop 26:41

Yeah, Jim, I've seen, I'm seeing some great questions in the chat room coming up. And so I think, Asplund, I want to run this first one by you because I think it's a great example. In the table in the report, you see some themes have higher ratios than others -- Woo, for example, at .82. Does that mean that Woo is more likely to be consistent in somebody's top -- in their, in their results than the average theme?

Jim Asplund 27:08

Yeah.

Austin Suellentrop 27:09

Yeah. So, so when you see certain themes are more likely to be consistent with people over time than other themes?

Jim Asplund 27:17

Yeah.

Austin Suellentrop 27:18

Right. So I think that's a good callout -- and I think Nate was the one who asked the question -- that yeah, there are some themes that when we, when you have them, those themes of talent, they're a little less likely to change over time, and other themes that are more likely to change.

Jim Asplund 27:31

Right.

Austin Suellentrop 27:32

And then I always love this question, Jim, because I ask this of you. You, you help me with my language on this a lot. The change from .7 to .73 -- is that statistically significant?

Jim Asplund 27:44

Well, it's two different samples and very different populations. Technically, the answer is probably "Yes." I think meaning -- there, you know, so the danger of asking a statistician, you know, you get a different answer than when you asked -- a different question than the one you asked. But it's, yeah, it probably is. I didn't even bother to check because to me, they look close enough to the same that I'd kind of figure what's the point, but probably is.

Austin Suellentrop 28:13

OK.

Jim Collison 28:14

Well, you said, Jim, you had said your answer. I had said "No" in the chat room. So I made a mistake. But you had said, Well, it went up, you know, in the sense that it's a, it's a, it's a small move. It's a, it's an increase. I don't know if I would necessarily, you know, make a big deal about it from that standpoint; I think that's kind of what you're saying. But it did.

Jim Asplund 28:34

Yeah, that's -- look, the technical answer is, I didn't check. But my guess is Yes. But I don't care. Because I kind of view them as the same. And, you know, meaningfully, you know, the real-world meaning of those differences, if they are different, is nominal. So --

Jim Collison 28:53

That's, and I think that was my point. I think if we try to start splitting hairs over that or making a big deal about it, it isn't something I wouldn't necessarily want the coaches necessarily going out and just saying, "Look at this! Look at the change!" You know, that -- in the, in the skeptic's world, in the math world and the folks that do this for a living --

Jim Asplund 29:10

Yeah, the real answer is better; was that it didn't really change that much, just reaffirming the previous result. And, and so, you know, in science, you like replication. And in here I kind of view that as, overall, a replication of our earlier result. We had a bigger sample this time. The previous one, by the way, was from our panel, which is representative of the U.S. population. So it was, it was a U.S. population. And this one was global. But you know, not designed that way. It was a convenience sample, but --

Jim Collison 29:43

Jim, did we look into -- so let's say I take it in one language and then the second time take it in another. When, when that is done, is there a statistical correlation of accuracy when done in the second language? So in other words, maybe I took it in English the first time, and then in German the second. Did we look into that at all?

Jim Asplund 30:02

Yeah, yeah, there's a table on that in there -- I think. I can't remember what I put in there at this point; I rewrote it, like, 5 times. The, this is the danger of doing this months after I finished it. Yeah, we did. And in fact, anybody from the Netherlands on the call, you should be proud. Like, 30% of the Dutch respondents on the second time actually took it in English the first time, or German. And so, you know, they -- and their reliability is still actually quite high. I, my, anecdotally, I typically see more of that either in the student population, or if you're working for a multinational and, you know, you're, you're used to doing everything in English usually is the first one they did, and then went back to their first language.

Jim Asplund 30:46

You know, maybe the, you know, depending on when it was, they may not have even known it was available in Croatian or Hungarian or Dutch or whatever, but then they took it. And so we have a pretty good sample of a few language pairs there. But basically, you typically see a little attenuation. So you know, as you might imagine, people's understanding of those languages isn't equal. And the translations are very good. But that may not be the same as your personal translation of it. So, but the effects are pretty small. So, so that's that's heartening, I think, if you, if you are in that situation.

Jim Collison 31:24

We also find local dialect has an effect on the way people read things. We even have that in English. A lot of, a lot of people that say to me -- I see this real-world -- like, "Oh, that translation's bad." And then I say, you know, we still struggle with language, even in English, of people understanding what this word means versus what this word means and so --

Jim Asplund 31:45

We do have a King's English version for the U.K. for that reason, reason, right? So --

Jim Collison 31:49

Yeah, yeah, no, there's, there's differences -- and even regionally here in the United States, we have differences in the way we use, the way we use words. Jim, in our -- oh, go ahead.

Jim Asplund 31:59

I was going to say, and language ages, by the way. So one of the things we do as we look through this is to see when we need to update items, because the colloquial phrasing is either there or not there in those things. And so, and as you might imagine, we, we're conservative with that, because that domino, then you have to re translate all those things. But sometimes you run it. And we've done that over time, you know, minor changes to things because it's kind of an old form of a word or a phrase.

Jim Collison 32:25

We say, Oh, we think like that doesn't -- would that even happen in our lifetime? Would words lose meaning? And we use a word, like, there's a word like called Pollyanna, right that we have used, we've used before -- someone who sees the positive side of everything. Well that, like, nobody 25 or below knows what that word means. And so that has left the vernacular in, in a lot of cases, and it's left quickly, like in the last 15 or 20 years. It has left very, very fast. Jim, question from our Facebook group, earlier in the week they wanted me to bring in, and we could talk for an hour on this. But in, in a, in 2 minutes, nature versus nurture. When we think about these talents, I know, no, I know. But I told them I would ask you, OK, so I told them I would. How much -- give us the quick "elevator pitch" on this. What's, what, what do we say -- nature versus nurture on talent?

Jim Asplund 33:17

Well, you know, we don't have a lot to go on from our own data, right? So unless we have, you know, 10,000 twins out there listening that want to sign up and help me do this, that would be great.

Jim Collison 33:27

Oh, don't, don't say that, Jim. They'll actually, people will come out to do it.

Jim Asplund 33:32

That'd be -- that's cool! Go recruit them! I want those people to take it!

Jim Collison 33:35

Keep going.

Jim Asplund 33:38

If you look at the literature, kind of the developmental process, you know, people have done research on twins. I live in Minnesota and there's a real big and longstanding excellent sort of twin studies, set of twin studies going on at the University of Minnesota here. They do some really cool stuff. Some of them have twins reared apart, if you've ever seen those papers or seen those news stories, they're fascinating. So people try to estimate, you know, the sort of heritability of some of these things. And so in terms of personality work that they've done in those kinds of areas, you know, you get a fairly high heritability, depending on the population. But it's -- at the end of the day, it's, it's hard to separate them, even, you know, in a controlled study, so you don't really know.

Jim Asplund 34:24

I mean, so I could give you a bad segue. So in grad school, one of my jobs was working for a bunch of research studies looking at aging, mortality and morbidity studies. So biostatistics of kind of, and we had twins, twins, there's a twins registry in Denmark. It's very old. And so we got data from, from that and so we were looking at data from that. We had a guy who was breeding fruit flies, so he'd, whatever the 1,000 analogue is of twins, you know, so 1,000 genetic, genetically identical fruit flies. Small weird thing: the laziest fruit flies live longer than the active ones. I thought that was really interesting. Some sort of oxidation problem.

Jim Asplund 35:04

Anyway, you look at kind of the the, and this is a purely a mathematical exercise for us to try to understand the models, that, that would show variability in aging, for a lot of reasons. And, you know, the rough rule of thumb was always kind of like you get done, and you go, Well, looks like it's about half and half, on something like that. And so my rule of thumb going into this is, absent any information, and we don't really have it, would be probably about half and half. If you look at the personality literature, you know, there's -- it's anywhere from that to less than that on some of the bigger studies I've seen recently. So there's a fair chunk of this that's probably heritable, that's probably hard to disentangle from, you know, growing up in the same house, if you're talking about siblings or children. But it'd be a fascinating bit of research to do if we had the data to do it. I wish, I wish we did.

Austin Suellentrop 35:56

That sounds to me like a challenge that we will have to put on the roadmap for the future of research at Gallup of figuring out how do we, how do we study that more? Because I'm asked the same question, Jim and Jim. I'm asked that quite a bit. And the reality is with our assessment, we don't have enough data points of twins to, to be able to pull this apart and do our own research on.

Jim Asplund 36:18

But you don't need twins, right? You know, you could do siblings and parent-child stuff. You need more data, and the assumptions you make are not as strong as you would with twins. My grandma had an identical twin sister, and they were spooky similar, even in their 90s. And so it was, you know, I get, you know, what people are thinking but -- my sister and I call each other at the same time all the time, and we're not twins. That stuff happens, but, it'd be fun to do, but we don't know.

Jim Collison 36:51

Austin, other questions that you have that you wanted to cover? I just jumped in. Sorry.

Austin Suellentrop 36:56

No, it's all good. It's all good. I think the, there was one question I want to clarify I saw in the chat room, which was about if somebody took the, took the assessment the first time and they didn't like their results, right. And they wanted to retake it because they didn't like their results. I would classify that in that bucket of times I wouldn't encourage them to retake it. Because when we think about why, what would happen the time somebody took it again, if they didn't like their first results, then there's gonna likely be some sense of intentionality in how they're answering it the second time to intentionally answer it differently. And so I think that, that that can potentially skew the results. Jim, I'd love your commentary on that.

Austin Suellentrop 37:39

But I think the bigger opportunity is that discussion and conversation with them initially as to what about the results they don't like? I think the most common driver of those results is a misunderstanding of the themes, is seeing something like Harmony and, and thinking of it as being soft, seeing something, seeing something like Belief and not knowing what the words mean. There, so I think oftentimes people's knee-jerk reaction when they don't like their results is because they're not fully understanding what they mean. Jim, anything you'd add on that?

Jim Asplund 38:12

Sure, I think you start with the dialogue, because I think there's a couple different kinds of "I don't like my results." One as I don't like them; one is, they don't seem right. And, and part of that, you know, the overlap and the reasons, which you gave, which is, you know, the, you put a name or label on things and people kind of grab the -- you know, we've changed the names of themes in the past because, you know, people didn't want to be told they were cautious. You know, that was a word they didn't like to hear. So, ta-da, you're Deliberative. And, you know, I mean, it's, and that's fair. I mean, I think, you know, those names have meaning. And so you just need to think about that when you're putting names and labels on these things. And, and, and who's to say -- we might change something again; some word might acquire a connotation that isn't right, and we don't want that to happen.

Jim Asplund 38:59

So there's, there's some legitimacy in that in the sense that, you know, you help people process what that means. You know, sometimes it's a fairly, you know, surface thing where, you know, I'm in strategic planning and don't have Strategic in my Top 3 themes. That's, that's terrible. And so, you know, people need to kind of work through. Sometimes, though, I mean, at the end of the day, the assessment shouldn't tell you who you are; it should help you explain it and help you understand it. But, you know, I've met people who look at this, and say, "That's just not me," and they could kind of make a case for that. Now, then, you kind of wonder, how did that happen? I mean, I think they answered the questions in a way that produced a result that's contrary to reality a little bit. But sometimes it's just a failure to acknowledge something about themselves that maybe they don't like -- that, you know, I've been told that I'm bossy and I got Command and, and, you know, that didn't really feel good. And that sort of thing.

Jim Asplund 39:52

And so I think part of it is helping people process you know, what it really means versus maybe, and or, you know, really means, you know, what it means, you know, in addition to what they think it means.

Austin Suellentrop 40:03

Right. Right. Wonderful!

Jim Collison 40:08

Jim, anything else you'd add to the report? You've spent years on it. What --

Jim Asplund 40:15

I waited years to have the data to do it. You know, cause it's funny, you know, just so people wonder, how can it take you so long? You know, because I've met lots of people who took it more than once. That still doesn't mean they're easy to find. So, you know, we didn't build our database so that you could spot them easily. And so it became kind of a detective game. Some people are easy to spot, and then you got to weed out people like me, who've taken it 25 times. And, and, you know, because we're testing it, you know, there's a handful of that. So there's cleaning and things like that. But, you know, it's, it's the, you know, our database isn't very good at tracking you, which I think you should be pleased to hear. And so it became difficult. I'm sure there's way more people than the close to the 58,000 people in here who've taken it more than once. Those are just the ones I can find. And so that's why it took a while.

Jim Asplund 41:08

But I think, you know, the, we've kind of covered the high points, I think, you know, the, the, the, you know, just to sum it up again, I think, you know, it's, it's very stable. It looks a lot like other real high-quality assessments, like the 5 Factor model-type assessments in terms of the way it plays out over time. And so it, it looks like a really good assessment should look.

Austin Suellentrop 41:30

Yeah. Jim, I want to, I want to thank you, Jim Asplund, for spearheading this, leading that, for always thinking about what we need to be researching, and leading the charge there. My time as a client, before I joined Gallup, I always took pride in knowing that I had Gallup's research engine behind the work I was trying to champion inside my organization; that I always had, I had the name and the brainpower of the team at Gallup to make sure that what I was presenting to my stakeholders was, was vetted and was trusted and was legitimate. And I'm really proud to take this research to the world because it's a big step forward in getting specific, in getting nuanced, in getting detailed around the questions we get asked. And I feel really comfortable and really confident in answering questions with more, with more nuance, and backing it with research rather than just with experiential background or just with theoretical background, but that we have research to justify the statements we're making.

Austin Suellentrop 42:30

And it's really, I hope, to all of our coaches out there, that you know, this is the kind of thing we take pride in at Gallup, we will continue to look at over time. We'll continue to research things. We'll continue to make sure we have the appropriate, as you said, phrasings of items so that they are relevant and are effective in measuring these things. So thank you, Jim, for all of your work on this. Jim Collison, thanks for having us on again. And to the all the people who are -- all the coaches out there and clients out there who are listening, thank you for your support of the, of the cause of being part of the strengths movement, and for taking time to be here today.

Jim Collison 43:05

And a couple reminders -- to the coaches, your job is to ask why and not tell. And so, as they bring these reports to you, and they're, they're asking these questions, fire those back to them and say, "I don't know, tell me more -- like, tell me more about you." That's whenever I have somebody come to me and say, "I don't think ...," and I'm like, "Let's just talk about that a little bit. You tell me." And so there's some great opportunities to start conversations. Also a reminder that all of these resources are available out at gallup.com. So gallup.com/cliftonstrengths/ -- no, it's gallup.com/cliftonstrengths at all -- is all it is. You can get there. I was thinking "/live." That's where the live page is, if you want to join us for these live programs. I've also been posting the link to the Resources page, it kind of looks here, I'll show you kind of, kind of looks just like this, if you want to head out to that, to that page. I could say it, but then you would never get there because it's a weird URL. We'll post it in the show notes as well. But look for this report: Stability -- The Stability of CliftonStrengths Results Over Time. And we'll make that available. It's available actually right now if you want to head out and take a look at it. Don't forget, you can join us and listen to this as a podcast. Just go to any podcast player and search "Gallup Webcasts." You can then get Called to Coach or Theme Thursday, both of those available out there as well. While you're on gallup.com, don't forget to sign up for the CliftonStrengths Community Newsletter. Austin and I spend a lot of time each month thinking about the content that goes in those newsletters for you. And if you sign up, we'll send it to you automatically each and every month. If you have any questions, you can email us: coaching@gallup.com. If you want to follow the live webcast schedule, go to gallup.eventbrite.com and just create an account, follow us there. If you attended the Summit, the 2020 Summit and, and, you still want to go back and listen to all the great recordings that are there. They're available right now. So head out to gallupatwork.com. While you're there, you can actually get registered for the 2021 Summit, which right now we're scheduling to be in-person. And I know that seems kind of weird to say that, but that's kind of what we're talking about. So again, gallupatwork.com. Best pricing is right now for the early bird pricing if you want to do that. Join us in our Facebook groups: facebook.com/groups/calledtocoach. And on LinkedIn, maybe you're not a Facebooker, if you're on LinkedIn, head out to "CliftonStrengths Trained Coaches" or search that on LinkedIn and you can join us there. I want to thank everyone for joining us live. We'll have a smidgen of a postshow at the end of this, if you are joining us live. With that, we'll say, Goodbye, everybody.

Jim Asplund's Top 5 CliftonStrengths are Analytical, Ideation, Individualization, Strategic and Maximizer.