Collecting feedback from learners and measuring how much they’ve learnt is reasonably straightforward if you simply stick to the basic Kirkpatrick levels. However, most L&D leaders have little time to devote to the more challenging task of gathering data on how learners are practically applying learning and the potential business impact. Fortunately, LEO Learning has found that analytics and big data can now make measuring business impact much easier and provide much more detail.
Kirkpatrick’s four-level model of training evaluation, based around reaction, learning, behaviour and results has been widely accepted as a valid blueprint for measuring learning impact but the problem is that this method is typically deployed as a one-off method of measurement.
By comparison, the new measuring techniques available focus more on how you approach data collection and analysis, aiming to make measuring business impact easier and more of a continuous business process.
LEO Learning’s research, carried out with our colleagues at Watershed and iVentiv, and involving more than 350 leading organisations across the world, suggests you should define your measurement strategy by asking three important questions:
- How does the learning programme relate to your organisational strategy? This includes considering the objectives and KPIs being addressed and what success will look like compared to your current situation
- How are your people currently performing? Are skills gaps responsible for any shortfalls, or are non-capability issues involved?
- What training is needed to meet the skills gap? Separate learner groups may have different needs, and the gap itself needs to be clearly defined
RECOMMENDED READING | 'Measuring the Business Impact of Learning: The Definitive Guide'
Getting started with measuring business impact
Simply getting started can be the hardest part of this highly valuable approach to measuring business impact, but answering these questions early on and having a clear vision of the level of performance your learning will achieve helps to align your learning design with your business strategy.
You might not know precisely what the ideal outcome will be, but working with management and other key stakeholders from the start and defining measurable metrics and KPIs will have results which are valued across your organisation. Forward planning, initiated collaboratively and in advance of the training commencing, goes a long way to seeing results from a measurement strategy.
The problem with traditional evaluation methods is that they tend to rely largely on experimentation, separating similar groups of learners into two groups and changing one aspect of one group’s experience to see if any meaningful impact is made. If two suitable groups can’t be created or compared, the experiment might be carried out by changing a variable part of training for a single group and attempting to measure the response.
LEO Learning’s research on new methods, though, shows how organisations are now able to leverage data to professionalise their activity in increasingly creative ways, while harnessing big data approaches to keep up the rapidly developing need for accurate reporting.
As an example of using this kind of data inventively, Watershed discovered that results on one formal course improved when a group of learners were accidentally told to take some modules in a new order, despite the course being designed in what subject matter experts considered the ideal order.
Leveraging data to measure learning performance
By collecting data in a learning analytics platform, you’ll have the ability to identify improvements as you go, as well as being able to model the likely impact as you change aspects of your training.
Where traditional learning evaluation based its data-gathering around answering one specific question, analytics and big data will give you a range of more in-depth, precise points to think about. You’ll learn more, for example, about how your learners use any informal resources provided alongside your formal programme, which will help you to improve your offering.
While your aim should be to gather as much learning and business performance data as you can, you might have drivers (such as measuring business impact) rather than a set question in mind. Any investment in automation and data analysis tools will be easily offset by the time savings made when collecting and analysing the data.
Bear in mind that the patterns you observe in your learning are starting points to think about. They might not always be significant, particularly if you are working with a large data set. As you understand more about how your learners use your programmes and resources, you’ll be better informed to make decisions about design and strategy, as well as the impact certain interventions and techniques have.
Traditional approaches to measuring business impact are not without merit: taking some action is more likely to improve your learning programme than doing nothing. Taking the first step into data and analytics can feel daunting, but a big data approach to measuring impact offers far more robust and adaptable ways to truly measure the impact of learning.
Get in touch with a LEO Learning expert to talk about measurement and approaches to data.