Measuring the deep impact of Immersive Learning

Measuring the impact of learning experiences can be a challenge!

When employees attend learning interventions, it’s generally expected that they will improve their job performance somehow and that this increase will ultimately have an impact on business performance. However, getting reliable data post learning intervention can be a challenge for Learning & Development (L&D), which is one of the reasons why the data in Emerging World’s latest CISL Impact Study is so powerful.

Corporate International Service Learning (CISL) experiences are a form of Immersive Learning in which participants step out of their comfort zone and travel across international borders to apply their skills and expertise on real-life issues that have a social impact.

Highly visceral in nature, CISL experiences enable participants to learn skills, develop new behaviours and foster a new mindset that helps them to make the critical shifts required to navigate in our complex world.

New data from Emerging World’s 2019 CISL Impact Benchmark Study supports this claim with long-term impact data on these experiences at all the levels on the Kirkpatrick framework.

Many in the L&D space are familiar with the Kirkpatrick model, a leading framework for evaluating the effectiveness of training. Given the model was invented more than 50 years, it is a testament to its influence and prevalence as a ‘go to’ tool for measuring learning outcomes. It considers the value of any type of training, formal or informal, across four levels:

• Level 1 Reaction evaluates how participants respond to the training
• Level 2 Learning measures if they learned the material
• Level 3 Behaviour considers if they are using what they learned on the job
• Level 4 Results evaluates if the training positively impacted the organisation

(https://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model, 2016)

Getting data at level one is reasonably straightforward with the administration of a post-event questionnaire but retrieving data at Levels two to four requires increasing amounts of tenacity and sophistication. One difficulty is that in order to get usable data, time must pass after the training event for people to have genuinely learnt things (level 2) or changed behaviour (level 3) and had an impact on results (Level 4). Identifying and engaging the participants in completing some form of evaluation becomes increasingly difficult the further away from the training that this takes place and, as a result, many training programs lack quality data on their effectiveness.

Corporate International Service Learning (CISL) impact participants profoundly. They take participants across international borders to learn skills, gain knowledge and develop new behaviours by challenging them to apply their skills and expertise on real-life issues that have a social impact. New data from Emerging World’s 2019 CISL Impact Benchmark Study backs up this claim with long-term impact data on these experiences at all the levels on the Kirkpatrick framework.

The 2019 CISL Impact Benchmark Study surveyed programme participants from eight global companies including BD, Credit Suisse, Mars, Microsoft and Salesforce. The 1115 respondents had all completed a CISL assignment at least 12 months prior to taking part in the Study which enabled the longer-term impact of these experiences to be evaluated. The results showed an enduring positive impact in a wide variety of areas including breadth and depth of learning which is outlined in the diagram below.

In the Study, the Kirkpatrick model is used to understand the long-term impact of CISL programmes on the Depth of Learning. When we talk about Depth of Learning, we are trying to understand how effective CISL programmes have been across the 4 levels of Learning.

CISL results to the Depth of Learning questions asked in the 2019 CISL Impact Benchmark Study.

It’s one thing to have the data on impact, but the CISL Study also goes further to identify factors that help strengthen that impact by analysing the data against a range of programme design variables.

Through this analysis we know that the top 3 impact levers for Depth of Learning are:

• Setting learning objectives
• Support on return
• Continued engagement with Partners

Setting learning objectives impacts depth of learning
The data suggested that participants that had set learning objectives before their assignment have a deeper learning experience. The chart below illustrates that when learning objectives are set there is greater impact on depth of learning. This is particularly strong for participants who agreed that changes in their behaviour had been observed by others and those who made a positive business impact on their organisation:

The data looks at participants that either agreed and strongly agreed versus disagreed and strongly disagreed that they set learning objectives prior to their assignment. These two sets of data were subsequently evaluated against five of the depth of learning measures.

Supporting participants in their return to work impacts depth of learning
The data also suggested that participants that felt supported on their return to work have a deeper learning experience. The chart below illustrates the significant impact that supporting participants on return to their role has on depth of learning. Those that do not feel supported on return are showing much lower responses for both changes observed by others and making a positive business impact:

The data shows participants that either agreed and strongly agreed versus disagreed and strongly disagreed that they felt supported in their return to work. Once again, the two sets of data were assessed against the depth of learning measures.

Continued engagement with partners impacts depth of learning
A strong alumni programme with opportunities for participants to engage with each other and to remain involved with their CISL programme and the partner that they supported can have a strong effect on the depth of learning. When participants continue to engage with the partners that they supported during their CISL assignment once they have returned to their roles, there is a higher impact on depth of learning. From the graph below, we can clearly see that when participants continue to engage with partners it has a greater impact on depth of learning across all measures:

The data shows participants that either agreed and strongly agreed versus disagreed and strongly disagreed that they continued to engage with their partner organisation following their assignment. Again, the two sets of data were assessed against five of the depth of learning measures.

Similar kinds of insight can be seen across other areas of impact and it all points to companies looking more closely at CISL as an approach to develop their leaders and being deliberate about how such programmes are designed to support good quality learning outcomes.

To find out more download our 2019 CISL Impact Benchmark Study which can be found here:

Download the 2019 CISL Impact Benchmark Study

kirkpatrickpartners.com (2016) [online] available at https://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model [accessed 30 October 2019]