Measuring typical performance using situational judgement tests

a&dc

a&dcA Senior Consultant from a&dc, Katy Welsh, examines the performance measurement of online assessments.

I’ve recently been working on the development of some bespoke Situational Judgement Tests (SJTs), in my opinion one of the most interesting and most challenging types of online assessments to develop.

Every time I work on such a project I have a quandary over the questions phrasing, as research has found there are differences between asking ‘What would you do?’ and ‘What should you do?’. This got me thinking more broadly about performance measurement in online assessments.

As an employer, the focus of individual assessment (whether for selection, development, or any other talent management), is likely to be on understanding the future performance of your employees.

Online psychometric assessments are often used at the beginning of a selection or development process as they are a cost and time-effective method of gaining some insight into the candidate or employee’s future performance on the job. Such assessments can be seen as providing a ‘first look’ at individuals, before moving on to more in-depth behavioural assessments.

Psychometric tests tend to fall into two categories: tests of typical performance and tests of maximum (or ‘maximal’) performance. Typical performance tests are concerned with assessing how people act in their day-to-day jobs.

They can be contrasted with maximum performance tests, which look at how an individual can perform when they are exerting maximum effort. Typical performance tests are concerned with what people will do, maximum performance tests are concerned with what they can do. [1]

Tests of typical performance include measures such as personality questionnaires and behavioural questionnaires. They have no right or wrong answers, but rather look at values, preferences and motivations.

Typical performance measures are generally multidimensional, in that they measure a number of different constructs (for example multiple personality traits). In contrast, tests of maximum performance are generally focused on one construct, such a ‘verbal ability’ or a specific technical skill. They will have clear right or wrong answers, and will usually have a time limit.

So which should you measure, typical or maximum performance? As with many things, it will depend on the situation and what you are trying to achieve. Are you trying to gain an impression of how the person will likely perform day-to-day, or do you wish to assess the limits of their capabilities?

A combination of both types of information is likely to be most useful. It’s helpful to understand how an individual will perform when they are in their ‘comfort’ zone, but managers and employers also need to know how to stretch their employees without giving them tasks that are far beyond their skillset.

On the flip side, it’s useful to know what an individual can achieve when they are working at their maximum, but it would be unrealistic to expect them to perform at this level all times.

The SJT is becoming increasingly popular as an early stage online assessment. This type of assessment can take many different formats, and can go some way to bridge the gap between typical and maximum performance.

Research has suggested that SJTs with the instruction ‘What would you do?’ provide a measure of typical performance, whereas SJTs with the instruction ‘What should you do?’ may predict maximum performance.[2]

While there is a link between typical and maximum performance, recent research has shown that this is not a strong enough relationship to simply infer one from the other[3]. Therefore, in order to understand what your employees are likely to do day-to-day and what they can really do when the chips are down, you should be using a range of assessments.

So going back to the SJT decision, I should be looking at question formats that complement the rest of the process, perhaps employing the ‘What would you do?’ format when used alongside an ability test, and ‘What should you do?’ when combined with a personality or behavioural assessment.

Katy Welsh, Senior Consultant, a&dc.

 

[1] Barnes, C. M., & Morgeson, F. P. (2007). Typical performance, maximal performance, and performance variability: Expanding our understanding of how organizations value performance. Human Performance20(3), 259-274.

[2] McDaniel, Michael A., Nathan S. Hartman, and W. Lee Grubb III. “Situational judgment tests, knowledge, behavioral tendency, and validity: A meta-analysis.” 18th Annual Conference of the Society for Industrial and Organizational Psychology. Orlando. 2003.

Stagl, K. C. (2006). The construct validity of a situational judgment test in a maximum performance context (Doctoral dissertation, University of Central Florida Orlando, Florida).

[3] Beus, J. M., & Whitman, D. S. (2012). The relationship between typical and maximum performance: A meta-analytic examination. Human Performance25(5), 355-376.

 

a&dc

About a&dc

a&dc are the leaders in behavioural assessment and development. We make a measurable difference to business performance by using behavioural assessment to recruit and select great talent into every job and developing leadership behaviours that enhance business results. Find out more.