Measuring training effectiveness and the ROI of localized L&D

In a survey on localization in L&D conducted with Training Industry, we learned that enterprises are localizing almost three-quarters of their training content. As I covered in another blog inspired by the survey, this – combined with other survey results – suggests that they recognize the value of localization in a training context.

 

It's interesting, then, that only 30% said that they evaluate the ROI of learning programmes by comparing outcomes by country, region or language (see chart below). It's a healthy minority, but does it mean that the majority are happy to accept the ROI of localized L&D initiatives on faith? That they don't know how to measure the impact of localization on training effectiveness? Or something else?

By their own account, most of these enterprises are measuring the ROI of their learning programmes. A full 70% are measuring improvements in employee performance, with 60% measuring learner satisfaction and 51% course completions. If they're already collecting and analyzing data relevant to any or all of these metrics, it's surely not a stretch too far for them to disaggregate their findings by country, region or language to gain insight into the impact of localization? Depending on what these enterprises are measuring and how, they might need to add localization-specific questions to surveys or be sure to collect data relating to the completion of localized courses, but this would be additional detail added to an existing process and competence. It's a matter of finding the time to make these changes – but time is often in short supply.

 

If you're looking at these survey results and thinking "we'd love to measure the impact of training on employee performance, if only we knew how" – you're not alone. Before I look at how you can do this, it’s worth noting that it takes time to build the lasting changes in skills and behaviour required to improve performance. No single training course is likely to achieve this in isolation. You should be measuring the impact of training initiatives over time and recognizing that other factors – such as management support, rewards and recognition, or the right tools for the job – also play a role in improving performance.

 

Nor is all training designed to improve employee performance. The aim might be (for example) to raise awareness, mitigate risk or comply with a requirement that has no performance impact. For these types of training, different metrics should be used to assess whether they have fulfilled their objective.

When improving employee performance is the aim of an L&D initiative – or the aim is to achieve a business goal that requires improved performance – then measuring the desired change in performance will clearly be a better indicator of training effectiveness than course completions or learner satisfaction. It's just that it's easier to count course completions or have learners complete a quick satisfaction survey immediately after training, than it is to measure their performance before and after training and link any improvement to the training.

 

To measure changes in performance and link them to L&D, you need to identify appropriate metrics – and be aware of their limitations. It's important to agree these before designing the training, not only because programmes will be more purposeful if you're clear about what they're trying to achieve, but also because you may need to implement new processes to collect and analyze the data.

 

Here are some examples of performance-related metrics and their limitations:

 

These are the ultimate results that improved employee performance is supposed to realize. Examples include growth in sales figures, higher customer satisfaction scores, fewer accidents, on-time delivery – obviously, they are highly dependent on the nature and purpose of the training.

 

Remember, though, that if an agreed outcome does improve following an L&D initiative, it doesn't automatically mean that training was the cause, or even part of the cause. Other factors may have been at play, or played a bigger role. You may therefore need to combine business outcomes with other metrics to tease out the impact of training.

 

These are inputs that are presumed to contribute to improved performance, such as an increase in knowledge, competence or confidence (assessed by tests before and after training) or a positive belief in the usefulness of training (assessed by a learning impact survey). Combined with business outcomes, these help to identify a likely causal link between training and improved performance.

 

Learning outcomes are more useful if assessed some time after training rather than immediately after. Then you can assess (for example) whether the learning has been retained rather than being quickly forgotten, or whether learners report that training has actually been applied in a real-world scenario rather than simply stating that they believe they will apply it in future.

 

So the challenge for using performance-related learning outcomes is whether you can create a robust process to:

As mentioned before, once you're putting in the effort to measure the ROI of training, it shouldn't be too hard to add localization into the mix. For example, you can assess the impact of localization on training effectiveness by:

Naturally there are different levels of granularity at which you can make these assessments. At the most granular, you could be assessing the quality of translation or the value of specific learning formats, structures or styles, for example.