Popular Does Not Mean Valid: Why Scientific Rigor in Personnel Selection Matters

The conversation with the candidate is part of almost every selection process. According to Diekmann and König, around 98 percent of the German companies surveyed use it (2015). It feels like information, and in the end you are left with an impression you trust. Research tests something else, namely how well a method predicts later performance. Between impression and predictive power there is a gap. This article shows how large it is and what it means for selection decisions.
Reading time: about 6 minutes
- Popular Does Not Mean Valid: Why Scientific Rigor in Personnel Selection Matters
- What research shows about predictive power
- Why the popular is rarely the valid
- What a type test promises and what it delivers
- The extraverted salesperson is not supported by the research
- What scientific rigor in selection means
- Conclusion
- Relevant Use Cases
What research shows about predictive power
Paul Sackett, Charlene Zhang, Christopher Berry and Filip Lievens recalculated the predictive power of common selection methods in 2022 (Journal of Applied Psychology 107, 2040 to 2068). Earlier estimates had set the values of many methods too high. After the recalculation the structured interview reaches .42 and the unstructured interview .19. Job knowledge tests come in at .40, empirically keyed biodata at .38, work samples at .33 and tests of cognitive ability at .31. Conscientiousness, one dimension of the Big Five, stands at .19, and years of work experience at .07.
Squared, these values show how much of the differences in performance a method explains (more in the article Predictive Validity). The structured interview explains around 18 percent, the unstructured interview around 4 percent and years of experience around half a percent. Structured means that all candidates get the same questions and the answers are rated against criteria set in advance. The same interview time then delivers more than four times the informative value.
Anyone who knows older figures will have seen different values. In 1998, Schmidt and Hunter reported .51 for the structured interview and .38 for the unstructured one, .51 for tests of cognitive ability and .54 for work samples. The recalculation lowered most values, most strongly for work samples and cognitive tests. The structured interview now leads.

Why the popular is rarely the valid
The gap between science and practice has been described for years. Benit and Soellner named it the scientist-practitioner gap in 2013. How it looks for personality questionnaires is shown by the surveys of Hossiep and colleagues: in large German companies, type-based instruments top the ranking. The figures are in the article Type Tests, the Big Five and the Barnum Effect. According to Diekmann and König, around 5 percent of the surveyed companies use tests of cognitive ability and 15.1 percent use personality tests. Tests of cognitive ability reach .31, conscientiousness .19.
Simple methods win out because they are quickly explained, their results are easy to remember and the people involved recognize themselves in them. That creates acceptance, and acceptance is easily mistaken for quality.
What a type test promises and what it delivers
Anyone who sorts candidates into types expects a statement about who fits the job. The type is supposed to justify the decision and make it more objective.
What it delivers first is a feeling of a hit. Martin Kersting describes in Personalmagazin 9/2015 ("Das beschreibt mich und andere") that statements that are true of almost anyone feel personal. The recognition is real and says nothing about whether the test measures what it claims to measure. In 1989, McCrae and Costa found no evidence for genuine, qualitatively distinct types (Journal of Personality 57, 17 to 40). Pittenger summarized in 2005 that test scores show one peak where genuine types would produce two, and that 35 percent of participants received a different four-letter type after four weeks (Consulting Psychology Journal 57(3), 210 to 221).
A type is a self-description. It records how someone is most likely to rate themselves, and it therefore predicts nothing about job performance or later behavior. The environment in which someone works does not appear in this description. Behavior can also be learned and, depending on the environment, shown against your own type.
For selection this has consequences. Anyone who screens out a candidate by type is, in more than one case out of three, screening out on a result that would have changed four weeks later. A general type is also not derived from the requirements of a job, while job-relevant assessment has to relate to the job and its function.
The decision looks well founded because a test result sits in the file. It is well founded only to the extent that the method predicts performance, and for the type model the empirical basis is missing. The company has a label, and the question of predictive power stays open.
The extraverted salesperson is not supported by the research
A common example of a label that carries decisions is the born salesperson. In the DISC model the red-yellow type counts as the ideal hire, in personality psychology the extravert. Adam Grant tested the assumption in 2013 with 340 employees of an outbound call center (Psychological Science 24(6), 1024 to 1030). He points out that earlier studies had found weak and inconsistent relationships between extraversion and sales success. In his data, revenue follows an inverted U: the highest revenue came from ambiverts with a medium level of extraversion, ahead of the strongly extraverted and the strongly introverted.
The meta-analysis by Vinchur, Schippmann, Switzer and Roth (1998) shows where the predictive power actually lies (Journal of Applied Psychology 83(4), 586 to 597). Potency, a sub-dimension of extraversion, correlates .28 with supervisor ratings and .26 with objective sales figures. A single facet therefore predicts something, and assigning someone to the type extravert does not support the assumption of the born salesperson.
For DISC we found no peer-reviewed study showing that high D and I scores predict better sales results. Anyone who claims this connection needs a study for it.
What scientific rigor in selection means
A clean selection starts with requirement criteria that are fixed before anyone holds an interview (see How to Create a Job Requirements Profile). Next comes a structured process with assessment that captures personality, values, motives or abilities, depending on the role sometimes several of them. The personal conversation comes last, and from there sympathy may decide, because the earlier steps have already filtered out the suitable candidates.
Scientific rigor in this sequence means that every step has evidence behind it. The requirements come from the job, the interview has structure, and the assessment has documented predictive power for the task. For methods and process there are standards, DIN 33430 and ISO 10667.
Conclusion

Popularity shows what companies are used to, predictive power shows what works. A type test works as an occasion for a conversation about yourself and carries no selection decision. The same conversation reaches .42 with structure instead of .19.
Relevant Use Cases
L8: External Selection for Unknown Leadership Candidates
A2: Apprenticeship and Trainee Selection
L3: Executive Search Support, CEO and Top-Level Selection
Which online assessment tools come into question for these cases, and how they differ, is covered in the PEATS Guides.
