7 Steps to Using Analytics to Improve the Evaluation of Learning
posted by Laurie Bassi
A recent CLO article reported that a growing number – nearly 60 percent – of CLOs are dissatisfied with their internal learning analytics capability. The article states that, “This reflects an ongoing trend: The state of measurement in learning and development is falling behind other areas of the business. CLOs are more dissatisfied with their organizational approach to measurement this year than last, continuing a trend of the past three years.”
In short, in most organizations a different approach is needed. More of the same (which typically means using Kirkpatrick levels 1-5, with an emphasis on the lower levels) won’t get CLOs where they want to be – understanding what’s working (and what isn’t) in learning and development initiatives and targeting resources at the most fruitful areas for improving business results. Instead, a more modern, “analytics-enhanced” approach is necessary.
Here’s what we’ve learned about how to transform an organization’s learning analytics:
1. Create an “authentic” learning impact evaluation by embedding it in a more holistic framework.
The fundamental problem with traditional (Kirkpatrick) learning evaluation is that it’s done in a vacuum. While some Kirkpatrick evaluations definitely have their place, a more authentic way to evaluate the impact of learning is to embed it in the larger context of measuring the overall management and development of a company’s people. In particular, companies should seek to identify opportunities to learn from “naturally occurring variations” in people’s work, learning, and leadership environments. All of those variations – not just learning-related ones – should be used to assess and predict variations in business outcomes. This requires that the CLO’s office collaborate with – or take the lead to create, if necessary – an analytics “center of excellence” within the organization.
2. Stop waiting for the perfect data warehouse. Instead, create a “data hut.”
At their core, data warehouses are designed for reporting – not analytics. So the sooner you realize that your organization’s data warehouse (whether current, pending, or hoped-for) is never going to enable you to do the learning analytics that you need, the better off you’ll be. To undertake analytics, you need to start by putting together heretofore disparate pieces of data (see Figure 1). This will enable you to do the following:
- Take a big step toward embedding your learning evaluation in the larger context of evaluating people management and development (step #1 above).
- Analyze why you are getting the impact that you are getting (not just what the impact is).
- Produce actionable insights about what levers to pull to create better business results through learning.
Figure 1. Build a HR Analytics “Data Hut”
3. Don’t let the perfect become the enemy of the good.
Generalizing #2 above, in the early days of creating an analytics-enhanced learning evaluation, you will almost certainly not be able to obtain all of these disparate pieces of data and integrate them into a unified analysis file. But don’t let that become an excuse for inaction. Start by putting two of these pieces of information together – for example, data from your LMS with employee engagement, or turnover data, or customer satisfaction data, or sales data. The important point is to begin to make progress, rather than to continue with the (unsatisfying) status quo.
4. Choose your initial analytics project carefully.
The best place to start is with a burning business issue. Examples might include one or more of the following:
- Customer satisfaction problems
- Lackluster sales
- High levels of regretted turnover
- Failure to achieve diversity and inclusion goals
- Stagnant or declining employee engagement
Design your initial analytics project to provide actionable insight on issues that are front-and-center for senior executives, and you will find yourself in a much better place.
5. Start under the radar.
Making progress towards better, more powerful, analytics-enhanced learning measurement does not have to take a lot of time or money. Choose the right initial project (#4) without fanfare. Just go do it. Use the findings from your first project, and the enthusiasm that it will engender, when properly presented (see #6), to bootstrap up your learning analytics budget and capability.
6. Remember: insightful reporting trumps data dumps.
Learning analytics is both a science and an art. The art comes in how you present the findings from your analysis. Less is often more. So focus on what executives need to know to drive better business results, and avoid the temptation to share every nuance and cool thing you might have learned in the course of the analysis.
7. Use learning evaluation to improve the effectiveness of learning.
It will also be tempting to use your newfound analytics capability to prove that “learning is working.” It’s far better, however, to use it to develop actionable insights for improving the business impact of learning. Doing so will generate significant returns in terms of the enhanced credibility and support that learning enjoys within your organization.
(This post was sent this month via email to our monthly newsletter subscribers. Click here if you’d like to subscribe.)