Big data: HR needs to stop reporting and start predicting

Big data can help HR influence business decisions

Big data offers HR some major opportunities to increase its strategic influence within the organisation and add value to its processes at all levels, by delivering predictive analytics.

Big data might seem like an HR buzzword, but it is one that will not go away – and one that HR should not ignore. The phrase refers to the huge volumes of data being generated in the modern world, and how we use it. Helping HR use big data to its advantage was the topic of a recent webinar from US blog Fistful of Talent, hosted by HR tech bloggers Steve Boese and Kris Dunn.

To harness big data, HR needs to change how it collects data, and to hire more data specialists, said Dunn. HR must also change how it uses data – it needs to stop reporting and start predicting. The biggest opportunity that big data offers to HR is in predictive analytics around high-volume, repeatable processes, such as recruitment.

Dunn and Boese offered practical suggestions for ways in which HR can start to use big data right away.

First, HR must identify the data it has. A good initial step is to draw up a schematic of available data on former, current and potential employees (within the boundaries of data protection legislation, of course). This creates the necessary volume of data to enable data modelling.

Second, make sure the data is relevant to the target audience. What metrics do the senior executive team need to see?

Third, scoreboard the data. Create tables that break down what big data has to say about how specific departments are performing, and are likely to perform. Fail to do this and the view of HR will never change, says Boese. To get the organisation to care about data, set up scoreboards and be ready to use the data they provide – against people, if necessary.

Turnover data is an ideal starting point for HR big data use, says Boese. Traditional turnover reporting is backward-looking. HR can take turnover data to the next step by annualising the data it already has to produce predictive analytics, projecting turnover for the coming month and the coming quarter. Employee age and tenure are likely to emerge as the two key factors influencing turnover. HR can use big data here to create a “relative risk of turnover” score for all departments.

How HR can harness big data

Accessing data should not be a problem for most HR departments, said Dunn. HR has plentiful access to data, for example payroll records, or data on absence and staff turnover. It needs to be more purposeful in collecting this data.

To capitalise on big data, HR needs more data specialists. There are two types of HR professional, Boese suggests: “the cop” (specialising in compliance and enforcement); and “the assassin” (agents of change and disruption within HR). HR is conservative by nature, thinks Boese: around three-quarters of HR professionals are “cops”. He argues that “the assassin” is better placed to harness big data – but HR needs an even mix of both types if it is to function.

HR needs to consider the three “Vs of big data”, said Dunn: volume (the huge amount of data being generated); velocity (the need to analyse and act on big data insights quickly); and variety (the ability to handle data in a range of formats). Others also introduce a fourth V, veracity, which ensures that the data is reliable and accurate.

For HR to master big data, the profession needs to learn how to handle data in both structured and unstructured formats, he continued. Structured data is data stored in a system in a defined, orderly way (such as traditional absence records). Unstructured data is data spread across multiple media and lacking unifying structure (such as video CVs and social media discussions).

  • Michael Carty

    “How data is used in an HR Function is critical to its success,” argues consultant Tony Perryman in a LinkedIn debate on HR data usage [https://www.linkedin.com/groupItem?view=&gid=2900498&type=member&item=5893375520145448963&qid=29f2ef3e-868b-4dcc-a216-8f3adf710b36&trk=groups_most_recent-0-b-ttl&goback=.gmr_2900498] that has spun off from this article. “Having a full understanding of how a unit is functioning financially and being able overlay people data allows a sensible conversation about potential people issues,” he continues. I’d be very interested to find out how Tony’s comment resonates for any HR professionals reading the article above. Is your HR department confident in utilising people data in discussions about how business units are performing financially?

    Turnover was cited by Steve Boese as a key metric for HR to use in the creation of predictive analytics. By coincidence, New Zealand-based HR blogger Richard Westney has published a post on HR’s experience with turnover data, entitled The Tyranny of Turnover http://hrmannz.wordpress.com/2014/07/13/the-tyranny-of-turnover/. Richard’s post is a very good one, and makes for particularly interesting related reading for anyone who might be feeling undecided as to the value of turnover data. Please let me know how Richard’s post relates to your own views.

  • http://www.adamaxon.com Adam Axon

    Interesting article MJ, I agree entirely with the sentiments. HR would benefit in being more skilled when it comes to data analytics, although I think the same thing to could be said of most business units, so perhaps we can skip beating up HR too much for not already having these skills!

    None the less, there’s no doubt it’s an area for improvement. There’s a few aspects to data analysis that need to be factored in, first the ability to actually carry out the analysis and generate the numbers, and second the ability to make sensible inferences around what this data actually means.

    I’ve just come off a Semester of studying statistics in my Psychology qualifications, during which I also took a bit of a wider interest in how stats are applied in that particular field.

    Based on those two things, I think the first aspect is definitely something that could and should be a mandatory teaching requirement in all HR qualifications. Have one core stats unit that teaches various research questions (differences, predictions, associations etc) and related analysis techniques that go with them, then in each other unit tie in a stats component as part of the assessment, one that’s related to the topic. i.e. a topic on performance appraisals could have a component of the assessment linked to using Anovas to look for group differences amongst performance ratings across divisions.

    I know that I’ve been able to tie in what I learnt in my Psych stats class to the metrics Involved in my HR project work at my current engagement. It’s turbo charged the analysis I’ve been able to carry out.

    When it comes to the second point however, inferring meaning from numbers, simply learning the methods in a class room isn’t enough. You need real world experience and a strong understanding of the data you are actually analyzing, these are things that you can only get from getting your hands dirty out in the field, and thus will take a lot of time even if the analysis skillset is there in future HR professionals.

    In the interim you’re liable to get a lot of poor inferences done from the data, this could hurt the credibility of the individuals in question, and perhaps the wider field in general. Take a look at Social Psychology’s current battle for credibility due to failures to replicate findings from some staple social psych experiments. How do we compensate for that period of infancy?

    Which brings me back to prediction. There are so many unknown variables when it comes to predicting real world events, I’m concerned that the idea of predicting things from HR data could be portrayed to be a relatively simple activity. There needs to be a debate around what standards of rigour we apply to our statistics, do we want to hold ourselves to the scientific rigour of a medical discipline like Psychology, or is that too excessive? But if it’s too excessive do the methods we use to predict from HR data lack the robustness to actually be of any meaning?

    Once again you could run into the issue where HR professionals start predicting things and the results never match up to those predictions. The boy who called wolf springs to mind, and again you could argue that HR’s credibility is going to be hurt by carrying our poor quality analysis than none at all.

    My views are no doubt influenced by having tackled statistics from a position of robustness in Psychology over the last few months, but even still, when I look at what’s happening even in Psychology in respects to: misguided inferences from data analysis; hacking of findings to support desired outcomes; failure to replicate findings
    it makes me a little bit cautious about suggesting HR should just jump straight in and start spitting out data.

    Long response, and I may have rambled, but I hope it adds something to the debate.

    • Michael Carty

      What a phenomenal comment, Adam – thank you for taking the time to compose and share it here!

      I completely agree with your suggestion that the ability to carry out data analysis and generate the requisite numbers “could and should be a mandatory teaching requirement in all HR qualifications.” But your point that HR should not rush in “and start spitting out data” is also very well made.

      What do you (or anybody else reading this) think might be an effective way for HR professionals to gain the necessary experience in “inferring meaning from numbers” without running the risk of HR hurting its own credibility by “carrying out poor quality analysis”?

      • http://www.adamaxon.com Adam Axon

        I had a bit of a think about your last question overnight, two thoughts came to mind.

        - First, there is no shortcut to experience. Short term pain, is probably something that we’d have to accept. But we’d do well to really broaden our view of change to something a little more longitudinal. Real change is slow, and takes time, often generations. Look at where HR was say 30 years ago, big improvement. In our day and age though we want everything instant, and change is no different, it’s just not realistic though and it causes a lot of unnecessary frustration in practitioners I think. (i.e. a lot of HR is dead type dramatisations)

        If we accept that improvement in HR will take time, say thirty years for analytics to be where we want, then it’s easier to accept that maybe the first five years of that will be a small regression as people gain the experience that will power future growth.

        - Second, one possible alternative is seek to appoint data analytic staff from non HR related fields, and hope that their inferential ability is strong enough to cope with an unfamiliar data source. I’m really dubious as to whether this works though, as I think a big part of making sensible inferences is really understanding your data in the first place. Still, might work.

        Thoughts?

        • Michael Carty

          Thank you for another outstanding comment, Adam.

          Sobering to think that HR could be three decades away from being able to harness data effectively.

          An HR data mentoring or coaching arrangement could be a solution. Experienced data specialists could coach those with less experience to help build the requisite “real world” experience of utilising data in the workplace.

          But there is a Catch-22 aspect. Many HR departments lack sufficient numbers of HR data specialists to enable future generations of HR data specialists to evolve.

          Consequently, your suggestion that HR departments appoint data analytics staff from non-HR fields sounds compelling (although, as you rightly point out, that brings its own problems).

          Another option is for HR to reach out to Finance and other departments within the organisation that routinely make extensive use of data, to tap into internal pools of data specialists. Perhaps internal, interdepartmental data skills mentoring arrangements could help HR build the data analytics skills it needs for the future?

      • http://workforcedynamics.wordpress.com Paul Strickland

        The subject is briefly touched upon in the CIPD Level 7 qualification and has been for years. However it is just the basics of the basics and I would disagree with Adam about “that the same could be said of other business units”. This is a wrong assumption I’ve actually learnt more about Analytics away from HR. It has been said HR is 5 years behind with Analytics, I will say in some cases the gap is widening! You will have extreme trouble trying to fit Analytics into a HR qualification, it is a subject in itself, Also don’t go down the blind dead end alley of stats!

        Analytics is more than that, much more! What we do need in HR are proper Business Partners who have real business experience and not some bright eyed uni grad. The crunching of the numbers needs a real Analyst.

        • Michael Carty

          Thank you for a very interesting comment, sir. If as you suggest that some HR departments are actually falling further than five years behind when it comes to analytics, is there any immediate remedial action that you think should be taken? Would it be best for the CIPD to move beyond just covering what you memorably term “the basics of the basics” when it comes to data and analytics for HR?

  • http://www.adp-es.co.uk/ ADP UK

    A very interesting article and some great comments that reflect some of the insights included in ADP’s new report – ‘Big Data in HR: the big questions being asked’. Our discussion also focuses on the need to up-skill HR professionals to be able to handle, analyse and use data effectively, as well as ensure it is structured in the right way to be useful. Relating to the skills debate mentioned, Matt Stripe the HR Director at Nestle, told us that his organisation has introduced a team of big data specialists, whose role it is to feed data insights to the core HR team – so this could be one solution, as Adam has suggested in his comment.

    Another challenge we identified is business integration, which is crucial for HR big
    data to be useful. HR data is scattered across the organisation and its insights are valuable to every department. HR must therefore work closely with others to obtain a truly holistic view of performance, and then use this information for the benefit of the business as a whole. This could also help pool skills in terms of analysing and interpreting the data, as Michael has suggested in his comment.

    But despite its undoubted potential and the current hype, our report is clear that data is not the whole story and HR professionals must continue to maximise their experience and networks to understand what the data is telling them. Our panellists agreed that if they are able to do this effectively, they will further build their strategic
    influence on the boardroom and the business.

    Annabel Jones, HR Director, ADP UK

  • http://www.adp-es.co.uk/ ADP UK

    A very interesting article and some great comments that reflect some of the insights included in ADP’s new report – ‘Big Data in HR: the big questions being asked’. Our discussion also focuses on the need to up-skill HR professionals to be able to handle, analyse and use data effectively, as well as ensure it is structured in the right way to be useful. Relating to the skills debate mentioned, Matt Stripe the HR Director at Nestle, told us that his organisation has introduced a team of big data specialists, whose role it is to feed data insights to the core HR team – so this could be one solution, as Adam has suggested in his comment.

    Another challenge we identified is business integration, which is crucial for HR big data to be useful. HR data is scattered across the organisation and its insights are valuable to every department. HR must therefore work closely with others to obtain a truly holistic view of performance, and then use this information for the benefit of the business as a whole. This could also help pool skills in terms of analysing and interpreting the data, as Michael has suggested in his comment.

    But despite its undoubted potential and the current hype, our report is clear that data is not the whole story and HR professionals must continue to maximise their experience and networks to understand what the data is telling them. Our panellists agreed that if they are able to do this effectively, they will further build their strategic influence on the boardroom and the business.

    Annabel Jones, HR Director, ADP UK

    • Michael Carty

      Thank you for taking the time to comment here, Annabel.

      Excellent point that “data is not the whole story.” But it is nonetheless an increasingly critical area for HR.

      I would be very interested indeed to read the ADP big data in HR report, if possible. Could you please be so kind as to post a link to the report here, if one is available? I’m sure other readers of this page would be keen to read it.

  • Linda Watson

    Hi Michael, analytics are very important. I have worked in many organisations who gather basic analytics generally as a means to fulfil KPI requirements – so they can be used as self-fulfilling prophecies if you know what I mean. What I have seen done poorly is the interpretation of analytics. HR doesn’t collect enough variety of data and then doesn’t correlate the data with other data. I don’t believe that it is necessarily a matter of lack of skill in HR more that HR doesn’t have the money to throw at it or the time to collect the data. No excuse – it can be prioritised. However, I don’t believe that HR has the skills to use analytics as predictors nor trends. Definitely a growth area.

    • Michael Carty

      Thank you very much indeed for this thought-provoking comment, Linda. In your experience, how often does HR tend to use the insights gleaned from data analytics in a manner that might be described as a “self-fulfilling prophecy,” please? And in which areas have you seen HR departments tend to do this?

      I like your point thats there is “no excuse” for HR not to take the time to collate and analyse data, and that it can (and perhaps should?) be prioritised, and that data analytics skills are “a growth area for HR” – amen to that!

  • http://www.personneltoday.com/ Personnel Today

    Thanks for your comment Linda – Do you think more needs to be done in this area for students entering HR? Do we need to make sure the next generation of HR folk are data literate?
    (Michael’s on leave this week but returns Monday when I am sure he will continue this discussion…)

    • Michael Carty

      Thank you for holding the fort on this one while I was on leave, m’colleague… and great question there as to what needs to be done to ensure that HR’s next generation is data literate.

  • Steven Toft

    Good piece Michael.

    I agree, it would be great to do some of this stuff but a lot of organisations don’t yet have their data in good enough a state. It’s true that there is lots of people data around in organisations but it’s not always coded and categorised in a way that makes for easy reporting. You need the cops to make sure the data is in the right state before the assassins can do their stuff.

    The problem is, the assassins need to tell the cops what sort of data they need beforehand. Much as they may not like to do so, the would-be strategists need to get involved in the data improvement process to make sure what is being done is going to give the organisation what it needs.

    And there’s the problem. The people sorting out the data don’t have the big picture and the people who want to be strategic think that getting down and dirty with data is beneath them. OK, maybe a bit of a generalisation but you get the picture.

    • Michael Carty

      Great comment, Steve – and thank you for taking the time to compose and post it. Given the scenario you describe, it sounds as if Steve Boese was spot on in his suggestion, in the big data webinar reported above, that HR requires an even mix of “cops” and “assassins” if it is to engage effectively with big data. But that could be easier said than done…

    • Mike Kennedy

      Hi Steven,
      The assassins would benefit from thinking outside the silo – combine HR data with LOB data. Instead of torchering HR data until it confesses, squeezing blood from a stone or some other metaphor – combine sales+talent or call center+talent datasets. All of a sudden new insights emerge and things start getting interesting and valuable. Instead of attrition was 25% last quarter, it’s 75% of the people that left were top sales performers, which projects out to impact revenue by x. Wow, now the business sits up and takes notice.

  • http://www.garethjones.me/ Gareth Jones

    Ive rather flippantly said in the past that an organisation gets the quality of HR function that it deserves. Or at least wants. i.e. the quality of the function is a result of the value the organisation (Aka leadership) sees in it. The same goes for data and analytics. I think Steven nails it here. It’s pure fantasy to start talking like this without the basics in place and most organisations just don’t have the data in the first place, let alone the scope of what they are trying to do with it.

    Josh Bersin has some hard research and facts on this. Only around 4% of organisations globally have “predictive analytics”, the fourth and highest level of data capability. And only a slightly higher number have ‘strategic’, level 3. Below that and its basic hygiene/operational analytics. Nothing wrong with that, but its limiting.

    The other point that Josh stresses is that you cannot jump from basic operational data – level 1 say – to level 4. Its just not possible to miss out a step. And even if you are totally committed, have the quality of systems (theres your other problem – awful HR technology and systems) and take this stuff seriously, know what you need, what you want and can get the right people in place – data experts – it will take you a minimum, thats MINIMUM, of 2 years. I think Paul is right to suggest 5 years or more.

    So nope, hiring a bunch of ‘data scientists’ and throwing them at your average Oracle/SAP installation just wont cut it. All of our work is with FTSE 100 companies. A lot of what we do requires data from clients to help measure and prove the ROI of the investment. Guess what the response is nearly every time we ask for the basics in terms of HR data? “Eerrr. not sure we have that easily to hand”! These are some of the largest companies in the world ffs!!

    The last point id make is that its not about data, or big data, or even predictions. The power in this is in unstructured data – conversation transcripts, social feeds, emails, unstructured logs etc and correlating this with more traditional quantitative data. The blend and the scale is where the power is because thats what delivers the things that really matter in prediction – patterns. You cant predict the future, not even in HR! But if you can see patterns you can start to predict, with increasing accuracy, what might happen next.

    • Michael Carty

      “You can’t predict the future, not even in HR!” << A very good point, sir, and quite possibly a worthy title for a future blog post? Excellent drawing together of some of the key themes to emerge from the comments on this post so far, and thank you also for bringing Josh Bersin's fascinating research into the discussion, too.

      Given that your suggestion that even in a best-case scenario, we could be many years away from HR being able fully to engage with and harness big data, I'd like to put to you the question from my earlier response to Paul's comment: Is there any immediate action that HR can and should be taking now, in order not to miss out on the opportunities and advantages offered by data, analytics and (possibly even) big data?

  • Mike Kennedy

    Velocity is the rate at which the data is coming in. Strictly speaking HR data isn’t big data because unlike real-time marketing or operational data, people aren’t getting hired and fired every second. This makes HR data far more manageable.