The ‘sham psychometric test’ controversy

Psychometric testing has been in the news recently, and not in a good way. According to recent reports, the Department for Work and Pensions (DWP) has required unemployed people to take “bogus” psychometric tests, and threatened to take their benefits away if they refused to do so.

A Guardian report, and an accompanying video, go on to say that the test is a “sham” and will produce the same results no matter what answers are given. Some websites and blogs go further, saying, for example, that “this ‘test’ is a tool for abuse and psychological torture”.

It seems this particular questionnaire was developed as part of a pilot project put together by the Cabinet Office’s behavioural insight team, otherwise known as the “nudge unit”. This group uses two approaches that are surprisingly novel in governmental decision making. First, come up with ideas for small interventions to “nudge” individuals’ behaviour in a direction that both works well for them and saves the Government money. Second, carry out randomised controlled trials to see if these interventions actually work.

One trial was in job centres, where 2,000 jobseekers were randomly assigned to one of two groups – half went through the existing system and half went through a new process. For the second group, their first visit included a conversation about getting back to work; on every subsequent visit they were encouraged to make clear plans for the next two weeks, and if they were still unemployed after eight weeks they were given further help. The results showed that after 13 weeks, the second group were 15%-20% less likely to be signing on than the first group. The project has now been expanded to larger trials.

Criticisms

So, a success story? Not quite. The psychometric test that has come in for criticism was part of this project, used at the eight-week stage to help jobseekers identify their strengths, thereby making them more able to put their best foot forward in applying for jobs. It is short (just 48 questions), is completed online and outputs a report with the respondent’s five key strengths – what’s not to like? Plenty, unfortunately, as the implementation of this idea is less than perfect. There are a number of issues:

Presentation. Try the test for yourself here and see what you think, but the appearance of the questionnaire does not inspire confidence. It has a very basic look and feel, the title text does not quite fit in the space reserved for it, and there is the occasional grammatical error (such as “I always look on bright side” and “Try to find a new way to use them then every day”). This may seem a little picky, but if people don’t take the questionnaire seriously, they will not take any advice derived from it seriously. In the jargon of testing, it will have low face validity.
The questions. Several questions are a little strange, for example: “In the last month, I have been thrilled by excellence in music, art, drama, film, sport, science or mathematics” or “I have not created anything of beauty in the last year”. Some may not be easy to understand for people with lower levels of literacy or who do not have English as a first language, because of unusual phrasing or double negatives (“I hesitate to sacrifice my self-interest for the benefit of groups I am in” or “I am rarely as excited about the good fortune of others as I am about my own”). As these groups are likely to be over-represented among jobseekers, this may be an issue. Without some additional explanation or context, people will misunderstand the questions and the results will be less valid.
The report. The report lists the respondent’s five key strengths, based on their answers; potentially this is a very positive message. However, it is not immediately obvious that this is what the report is doing, and some additional text helping the reader to understand how they can use the results would be extremely helpful. Some critics have even suggested that people will get the same report no matter what answers they put down; this is, in fact, not the case and different reports will be generated from different answers – although, as the reports are very similar, this is not obvious.
The origins of the questionnaire. It has been reported in the press that the DWP test is a cut-down version of a much longer (120 or 240 question) “character strengths” survey published by the US-based VIA Institute on Character, and that VIA refused permission for the use of the shorter questionnaire, calling it a “non-validated version”. If true, this is very worrying, as it would mean that this new version of the test is not fit for purpose.
The way the questionnaire appears to have been used. This may be the biggest issue. Forcing people to complete an open personality questionnaire against their will (if this is indeed what has happened) is unlikely to have a useful outcome. Doing so on a large-scale programme in the full glare of publicity could be seen as asking for trouble. This is a real shame, because we know that accurate personality questionnaires, with specific and focused reports, can be incredibly useful in career counselling.

Can the positives outweigh the negatives?

Any personality questionnaire is a means to an end, but how it is used is just as important. Unfortunately, the world seems to have decided that the DWP questionnaire was designed from the outset to be used in a less than positive way.

Questions such as “is it reliable?”, “is it valid?” or even “could it be useful?” may be drowned out – and the questionnaire, for all its flaws, is part of an approach that does seem to give results.

Errors in implementation can be corrected and a new version trialled; this is a part of the whole ethos of evidence-based policy. Let us hope that the furore does not prevent this happening, and does not detract from the many places where psychometric tools are used in a positive way to widen and inform people’s career choices, to help them in the job search process, and to aid their development once in employment.

John Hackston is head of R&D at OPP Ltd