MCBASSI & COMPANY

How to Improve the Evaluation of Learning

avatar

We hope this finds you well, your new year off to a good start, and your resolve to make improvements and have a positive impact at work intact!

Maybe one of your New Year’s resolutions was to make improvements in the impact that your firm’s investments in learning have on its business results? Or maybe to improve your ability to evaluate those impacts? Perhaps even to elevate the strategic importance of learning? If so, we have a (belated) holiday gift for you!  Our (significantly expanded) ATD paper entitled “7 Steps to Using Analytics to Improve the Evaluation of Learning” can be downloaded here.

We hope this stimulates some fresh thinking and helps you to make progress on a problem that has long vexed our profession. Drop us a line and let us know!

(This post was sent this month via email to our monthly newsletter subscribers.  Click here if you’d like to subscribe.)

7 Steps to Using Analytics to Improve the Evaluation of Learning

avatar

A recent CLO article reported that a growing number – nearly 60 percent – of CLOs are dissatisfied with their internal learning analytics capability.  The article states that, “This reflects an ongoing trend: The state of measurement in learning and development is falling behind other areas of the business. CLOs are more dissatisfied with their organizational approach to measurement this year than last, continuing a trend of the past three years.”

In short, in most organizations a different approach is needed.  More of the same (which typically means using Kirkpatrick levels 1-5, with an emphasis on the lower levels) won’t get CLOs where they want to be – understanding what’s working (and what isn’t) in learning and development initiatives and targeting resources at the most fruitful areas for improving business results.  Instead, a more modern, “analytics-enhanced” approach is necessary.

Here’s what we’ve learned about how to transform an organization’s learning analytics:

1.  Create an “authentic” learning impact evaluation by embedding it in a more holistic framework.

The fundamental problem with traditional (Kirkpatrick) learning evaluation is that it’s done in a vacuum.  While some Kirkpatrick evaluations definitely have their place, a more authentic way to evaluate the impact of learning is to embed it in the larger context of measuring the overall management and development of a company’s people.  In particular, companies should seek to identify opportunities to learn from “naturally occurring variations” in people’s work, learning, and leadership environments.   All of those variations – not just learning-related ones – should be used to assess and predict variations in business outcomes.  This requires that the CLO’s office collaborate with – or take the lead to create, if necessary – an analytics “center of excellence” within the organization.

2.  Stop waiting for the perfect data warehouse.  Instead, create a “data hut.”

At their core, data warehouses are designed for reporting – not analytics.  So the sooner you realize that your organization’s data warehouse (whether current, pending, or hoped-for) is never going to enable you to do the learning analytics that you need, the better off you’ll be.  To undertake analytics, you need to start by putting together heretofore disparate pieces of data (see Figure 1).   This will enable you to do the following:

  • Take a big step toward embedding your learning evaluation in the larger context of evaluating people management and development (step #1 above).
  • Analyze why you are getting the impact that you are getting (not just what the impact is).
  • Produce actionable insights about what levers to pull to create better business results through learning.

Figure 1. Build a HR Analytics “Data Hut”

 

3.  Don’t let the perfect become the enemy of the good.

Generalizing #2 above, in the early days of creating an analytics-enhanced learning evaluation, you will almost certainly not be able to obtain all of these disparate pieces of data and integrate them into a unified analysis file.  But don’t let that become an excuse for inaction.  Start by putting two of these pieces of information together – for example, data from your LMS with employee engagement, or turnover data, or customer satisfaction data, or sales data.  The important point is to begin to make progress, rather than to continue with the (unsatisfying) status quo.

4.  Choose your initial analytics project carefully.

The best place to start is with a burning business issue.  Examples might include one or more of the following:

  • Customer satisfaction problems
  • Lackluster sales
  • Safety
  • High levels of regretted turnover
  • Failure to achieve diversity and inclusion goals
  • Stagnant or declining employee engagement

Design your initial analytics project to provide actionable insight on issues that are front-and-center for senior executives, and you will find yourself in a much better place.

5.  Start under the radar.

Making progress towards better, more powerful, analytics-enhanced learning measurement does not have to take a lot of time or money.  Choose the right initial project (#4) without fanfare.  Just go do it.  Use the findings from your first project, and the enthusiasm that it will engender, when properly presented (see #6), to bootstrap up your learning analytics budget and capability.

6.  Remember: insightful reporting trumps data dumps.

Learning analytics is both a science and an art.  The art comes in how you present the findings from your analysis.  Less is often more.  So focus on what executives need to know to drive better business results, and avoid the temptation to share every nuance and cool thing you might have learned in the course of the analysis.

7.  Use learning evaluation to improve the effectiveness of learning.

It will also be tempting to use your newfound analytics capability to prove that “learning is working.”  It’s far better, however, to use it to develop actionable insights for improving the business impact of learning.  Doing so will generate significant returns in terms of the enhanced credibility and support that learning enjoys within your organization.

(This post was sent this month via email to our monthly newsletter subscribers.  Click here if you’d like to subscribe.)

Training Magazine Top 125

avatar

Training Magazine recently announced its 2010 Training “Top 125” (PWC was ranked #1 for the third year in a row). 

This is always one of our favorite lists of the year – it’s one of the very few public sources of information on how much (some) specific organizations are spending on formal learning, so it’s always fascinating to go through the fine print on the new list each year.

Unfortunately, it looks like this will be the last “Top 125,” as Nielsen Business Media announced it’s shutting down Training Magazine and its companion website.

E-learning instead of slashing budgets

avatar

Employee training budgets are traditionally one of the first casualties of a depressed economy.  Slashing employee training, while providing a certain amount of immediate financial relief, can have negative long-term consequences.  A recent Training Magazine article explores ways that companies can effectively shift to e-learning, enabling them to reduce costs without sacrificing human capital development.