The Relationship Between Strengths‑Based Employee Development and Organizational Outcomes

Meta-Analysis, Hypothesis, Methods and Results

A meta-analysis is a statistical integration of data accumulated across many different studies. Meta-analysis has the potential to provide uniquely powerful information because it accounts for measurement and sampling errors and other idiosyncrasies that distort the results of individual studies. A meta-analysis eliminates biases and provides an estimate of true validity or true relationship between two or more variables. Statistics typically calculated during meta-analyses also allow the researcher to explore the presence, or lack, of moderators of relationships. It provides a method by which researchers can determine whether validities and relationships generalize across various situations (e.g., across firms or geographical locations). For the present analysis, we used the Hunter-Schmidt methods random effects model meta-analysis methods (Schmidt & Hunter, 2014).

Meta-analyses can be conducted on cumulative studies of the relationships between two or more variables of interest or the impact of two-group experimental interventions. The former are meta-analyses of r values whereas the latter are meta-analyses of d values (the difference between treatment and control groups divided by the pooled standard deviation). Meta-analytic mathematics, which uses advanced statistical methods such as reliability and range restriction distributions, are much more amenable to use of r values than d values. Since d values can be directly transformed into point-biserial r values, and vice versa, it is easiest to convert d values into r values, conduct the meta-analysis, and then convert the true score r values back into d values for interpretative purposes. That process was used for this study.

For this meta-analysis, we corrected for artifactual sources of variation such as sampling error, measurement error and range restriction, where possible. Measurement error was corrected in most dependent variables based on artifact distributions obtained for previous Gallup meta-analyses. Test-retest reliability estimates were used based on Scenario 23 in Schmidt and Hunter (1996). Scenario 23 takes into account that some change in dependent variables (stability) is a function of real change.

The most general definition of a Gallup strengths-based intervention is one where a respondent completes the StrengthsFinder assessment and is made aware of his or her top natural talents. In practice, strengths-based interventions vary in the objective, type and magnitude. In some cases, respondents are given more advanced coaching and training and in other cases, they are given more basic information such as a book or website description and tutorial. In some organizations, the interventions were designed for managers of teams, while in other organizations, individual contributors were given interventions.

Gallup researchers accumulated research studies comparing the intensity of strengths‑based development interventions by business/work unit. In some studies, business units that had been given a strengths-based intervention were compared to those that had not. In other studies, business units with a low (but non-zero) percentage of employees receiving a strengths intervention were compared to business units where a higher percentage of employees learned to develop their strengths. These studies included randomized experimental designs, but the large majority were quasi-experimental, utilizing wait list control groups rather than randomized treatment and control groups. Where possible, variables that were hypothesized to explain possible differences between non‑randomized treatment and control groups were utilized as statistical controls in analyses (i.e., baseline engagement, geography, business/work unit age, trade area market statistics, product type).

Researchers categorized the strengths-based interventions into four general types.

1. Business/work units included at least one person who completed the Clifton StrengthsFinder. Dependent variables were compared to business/work units where no one completed Clifton StrengthsFinder.

2. The percentage of individuals who completed the Clifton StrengthsFinder within a business/work unit was recorded. In this case, the treatment group independent variable could range from 1-100 percent.

3. An individual manager completed the Clifton StrengthsFinder along with a manager developmental course. Business/work unit dependent variables were compared to those of managers who had not completed the course.

4. An individual manager completed the Clifton StrengthsFinder. Business/work unit dependent variables were compared to those of managers who did not complete the Clifton StrengthsFinder.

Six general dependent variables were identified across studies: sales, profit, customer engagement, turnover, employee engagement and safety (accidents). The following is a description of each of the six dependent variable outcomes included in the studies.

• Sales: Sales, close rates, units per transaction, revenue growth, revenue per labor hours, sales in comparison to budget or goal, comparable sales growth and productive utilization

• Profit: Overall percent profit of revenue, profit increase, gross profit growth, margin erosion (reverse scored), margin versus target or goal, profit of existing customers, EBIT

• Employee Engagement: Business-unit-level average scores on engagement surveys

• Customer Engagement: Customer perceptions of quality

• Turnover: Annualized business/work unit turnover rate, first 90-day turnover rate

• Safety: Worker comp costs, workers comp incidents, patient falls, accident frequency, accident severity

Across studies, there was substantial variation in the proportion of the overall sample in organizations that were administered a strengths intervention. These values ranged from less than 1% to 99% (proportions of less than .01 to .99). With any proportion, the variance is maximized at .50. As such, departure from .50 reduces the possible effect size. Range restriction corrections were made based on an artifact distribution of independent variable estimates of U (sd/SD) across studies. Different artifact distributions were created for outcome-intervention combinations. In this case, correction for range restriction makes the size of true effect estimates more similar in magnitude to what one would expect in equally sized treatment and control group designs.

In an exhaustive review of Gallup’s inferential databases, organizations with both StrengthsFinder data and performance data were accumulated. Researchers limited their scrutiny to organizations with a minimum of 30 complete StrengthsFinder responses, and a few studies had to be removed due to lack of identifiable contrast groups. In the end, a total of 43 studies were conducted in 22 organizations and included 1.2 million individuals.

Study organizations came from a wide range of industries, including heavy equipment and vehicle manufacturing; retail and commercial banking; mass and specialty retail; electric utilities; finance and insurance; health care; aerospace; food and other agriculture products; building materials; investment services; education; and pharmaceuticals.

The total study population was geographically diverse as well, with business/work units from 45 countries. The number of countries per study ranged from one to 36.

The following steps were followed in conducting this meta-analysis:

1. Studies were categorized by type of outcome, type of strengths intervention, and whether the study utilized control variables or not.

2. d values from experimental and quasi-experimental studies were converted to r’s or point-biserial r’s, depending on the nature of the treatment effect variable (in one intervention type, the treatment variable was continuous—percentage of people within a business/work unit who were administered the Clifton StrengthsFinder).

3. Meta-analyses using artifact distributions were conducted, reporting observed and true score effect sizes, standard deviations and generalizability statistics.

4. r values were converted back to d value effect sizes.

5. Utility analysis was conducted to estimate the practical value of the effect size estimates of the various intervention-outcome combinations.

This study focuses on the relationships between learning or developing strengths and measures of organizational performance. Meta-analytic and validity generalization statistics for these relationships are shown in Table 1.

Mean observed correlations and standard deviations are shown, followed by estimated true validities, after correcting for dependent variable measurement error and within‑organization range restriction. This range-restriction correction places all organizations on the same basis regarding variability in the independent variable. These results can be viewed as estimating the relationships across business/work units within the average organization.

The findings show generalizability across organizations, as indicated by the 90% credibility values, all of which match the direction of the hypothesized relationships (Schmidt & Hunter, 1977). That is, Clifton StrengthsFinder completion effectively predicts these outcomes in the expected direction across organizations, including those in different industries and different countries.

For some of the measures, study artifacts explain most of the variance in correlations. For safety and sales, at least two-thirds of the variance in correlations is attributable to sampling error, range variation or measurement error. The results for profit measures were similar, but to a lesser degree; over half of the variability in these correlations is attributable to measurement artifacts.

In the case of customer and turnover measures, the sample of studies has much less variance between the effect sizes than would be expected by sampling error. This often happens with small numbers of studies per table entry, as was the case here. As a consequence, the estimated variance attributable to artifacts exceeded the total observed variability.

As noted earlier, variables that were hypothesized to explain possible differences between non-randomized treatment and control groups were utilized as statistical controls in analyses. As with the dependent variables themselves, the availability and quality of these control variables differed markedly both within and across organizations.

• Safety: All studies included control variables, including employee engagement, geographic identifiers, and employee and market demographic variables.

• Customer: Two of the three studies included control variables, including employee engagement, job and product types, and other employee demographic variables.

• Turnover: All three studies employed control variables, including employee engagement, geographic identifiers, manager tenure, employee tenure, number and type of competitors, and employee and market demographic characteristics.

• Engagement: All studies were controlled for engagement survey administration cohort (baseline engagement before intervention).

• Profit: Six of the nine studies used control variables, including employee and customer engagement, geographic identifiers, employee tenure, product type, employee and market demographic characteristics, business/work unit characteristics, and number and type of competitors.

• Sales: Seven of the 10 studies used control variables, including employee and customer engagement, geographic identifiers, employee tenure, product type, employee and market demographic characteristics, business/work unit characteristics, and number and type of competitors.

In total, 85% of studies in this meta-analysis utilized control variables of some type.

As noted earlier, the studies included in the meta-analysis used four different research designs. One limitation of the meta-analysis is that the number of studies per design was not large. Table 2 shows the range of different study designs used for the analyses.

For most outcome measures, there was significant heterogeneity in study design. Given the small number of studies per design type, it is difficult to draw many inferences regarding the influence of different types of strengths intervention, for example.

Acknowledgements

• Jim Asplund, M.A.
• James K. Harter, Ph.D.
• Sangeeta Agrawal, M.S.
• Stephanie K. Plowman, M.A.

The authors thank John Fleming, Ben Wigert, Kirti Kanitkar and John Reimnitz for contributing important research studies, client data and database information.