ITD Approach to Evaluation of Training & Return on Investment (ROI) & MIRTEX
The goal of evaluation
Whenever any organisation trains its employees it does so with an objective in mind. This objective can be extremely varied; equipping the people to do their job to the necessary legal requirement; giving ideas on how things might be done differently; opening minds to new ways of thinking; improving skill levels; as a reward for a job well done. With an objective in mind training is budgeted, designed, organised and implemented.
In assessing whether the training achieves the objective many organisations rely on the initial reaction of the participants at the end of the training, along with the observations of the training professionals who organised the training. Training professionals can be heard saying ‘as long as the participants are happy…’ hence the term happy sheets to describe the feedback forms completed at the end of most training sessions.
With some training an assessment of its success and impact is intrinsic. Training towards a qualification, for example, requires the participant to pass an exam or complete an assessment. These tests establish whether the participant has gained knowledge or ability. This test approach can be applied to other areas. For example, if a person is trained to use a software package they can be easily tested to see whether they have retained knowledge and can apply this knowledge in using the software programme.
Whether knowledge is actually used in the workplace is a different question. In the latter example we may be able to establish quite simply if the person has learnt the software programme, but we cannot predict if or in what circumstances they will use it. Moreover if we discover that this person does apply their new knowledge of the software programme, we may also wish to know whether this new skill has any positive impact on their work. Then we could go further by asking whether the use of this new skill has had any positive impact for the organisation.
Evaluation is linked to the objective for the training. In order to evaluate a clear objective is required. It is likely that budget holders are going to be increasingly required to justify and measure the effectiveness of any training investment.
The need to evalaute training
The need for evaluating any training is to ask the fundamental question, ’Is the money we are spending, well spent?’ Training professionals might rephrase this question to ask, ‘is training achieving the objective set for it?’ But business professionals are more likely to ask ‘what value does this investment bring to the organisation?’ When we start to explore evaluation we soon arrive at the question of, ‘what are we evaluating?’
So before we evaluate we perhaps need to ask why we are evaluating. What is the objective of the evaluation? And what is the objective of the training?
In this way a desire to evaluate training can easily be lost in a confusion of objectives. In order to clarify we can simplify evaluation by first summarising what is to be evaluated:
1. Has the person learnt anything from the training?
2. Has the person applied this learning in their job?
3. Has the application of this learning had any positive outcomes for the individual?
4. Has the application of this learning had any positive outcomes for the organisation?
Evaluating training could aim to establish if learning has taken place, if this learning has been applied and what impact this application has had.
In the current economic climate evaluation needs to establish if a learning objective has been achieved, but to be truly useful to business professionals with budgetary responsibility, it needs to establish the value to the organisation.
A UK transport organisation recently conducted advanced driving training for all its truck drivers. The training taught them how a different driving style can save fuel and make journey times shorter. The impact was a 20% reduction in the volume of fuel purchased.
So evaluation is useful to assess if the objective for the training has been achieved, but even better is to establish the value to the organisation.
[dottedline][/dottedline]
Is evaluation possible?
If the objective is to establish whether the training achieves a learning objective, this is fairly straight forward and the post course evaluation sheets can potentially do this well. For example, a group of managers have a need to increase their abilities in budgeting and finance. Training is implemented and the managers are asked; ‘did you learn what you needed to know about successful budgeting?’ Usually participants on training courses are clear on whether they have learned something useful or not.
Whether the managers will change their behaviour and in the example improve their budgeting accuracy is another matter. Achieving a learning objective might be reasonably simple, but can we establish the value of training to the organisation?
With some subjects this is more easily quantifiable. Negotiation skills training for lawyers can be followed up to discover if negotiation techniques learnt in training have been applied and resulted in better outcomes for the firm and for clients. For example one lawyer following an ITD negotiation skills programme said, ’the training taught me that in face to face negotiation silence can be powerful, so when in a damages claim the opposing lawyer offered £10,000 I said nothing and after a period of silence the offer was increased to £15,000. I could so easily have accepted the first offer on behalf of the client, but by staying silent we achieved a much better result.’ Examples like this demonstrate learning has taken place and has been applied with a positive result.
Sales training can be simpler to evaluate. Take the example where the need is for lawyers to develop a pipeline of business opportunities. Training is developed and implemented. Afterwards, evaluation can review the lawyers’ pipeline before and after the training.
Evaluation is possible but it tends to be complicated. Exploring and finding out how learning has been applied takes time and effort. Any evaluation tool needs to be sophisticated enough to be adapted to different training subjects and methods. It might be useful to have the same tool to measure software training as well as leadership training as well as stress management training. We may also want a tool which will be flexible in the information we require it to discover. For example we might want to discover if the participant learnt useful ideas in a training session, but we also might want to learn if they found the training motivational. Then we might want to explore how it has been applied and whether it had any positive results and if these results have any impact on the organisation.
One of the challenges with measuring the impact of training is to quantify the amount of impact directly attributed to training versus the amount attributed to other factors, if possible the evaluation tool needs to account for this or at least acknowledge these factors.
[dottedline][/dottedline]
Pre-evaluation of training
We are currently working on a methodology for pre-evaluating training. This involves measuring participant’s confidence in delivering on the training objectives before and after the training activity. From this we then hope to identify if we could predict which participant’s need extra support in helping to deliver the training objectives. Additionally it could help us identify if any participants might be better not to attend the training if there would be no chance of them delivering on the objectives. This methodology would also give us further data to support the argument that training needs internal support from line managers and others, for it to be effective, and not for it to be seen as an isolated event, with no follow up.
[dottedline][/dottedline]
The MIRTEX approach
MIRTEX is the Measure of Impact and Results of Training Index. 1 – 10.
ITD developed the MIRTEX evaluation tool following a conversation with a client, where the client stated, ‘What would be great is if we could say that this training scored 6 out of 10 or that training scored 9 out of 10, a bit like the Richter scale but for positive impact. The higher the score achieved the more positive the impact’.
The MIRTEX approach has the objective of providing a tool to evaluate and measure training investment. Training evaluation is inherently difficult and complex, especially when attempting to measure the impact of business skills training. Sales training can be a little simpler, if sales results can be used as a measure of impact.
The MIRTEX approach uses a telephone structured interview technique based on an evaluation scale.
Training participants are called between 1 – 4 weeks following a training course. The MIRTEX researcher uses a blend of structured questions with a flexible approach to find evidence of the impact of the training.
The evaluation scale scores the training from 0 to 10 where 0 is no impact and 10 is measurable financial impact.
The MIRTEX approach goes direct to the training participants and seeks evidence of impact. One potential challenge with this approach is that it depends on the reliability of answers from the participants, in other words, could the participants over state or under state the impact of the training. Clearly they could, but one way of reducing this possible weakness is in the ability of the researcher to ask searching questions that check statements and answers to previous questions. In addition statements can also be validated by seeking corroboration from their line manager.
The MIRTEX approach goes some way in collecting evaluation data which is both quantitive and qualitative.
It evaluates how well the training has performed in providing useful ideas for the participants, takes this further by discovering whether these ideas have been applied, still further by assessing the impact of this application and finally to measure any financial impact of the training.
[dottedline][/dottedline]
Case Study
Client:
A pharmaceutical business based in UK with all the typical office based functions including marketing, medical, safety, administration etc. The business also has a significant field based sales force.
Background:
The client holds conferences for the field based team and the office functions to get together to discuss and present the plan for the year ahead.
Need:
The client asked ITD to obtain data in order to answer the question; ‘did the last conference do what we wanted it to do?’
Activity:
We contacted 48 of the conference participants and used the MIRTEX scale and structured questions to evaluate the impact of the conference with each person.
Outcomes:
1. 90% of the participants responded to the telephone approach.
2. Data was collected from a number of people from each department in the business.
3. Qualitative data was collected on each part of the conference, specific activities and the impact it had.
4. The data collected gave the client exactly the information they required. The client was able to say with a high level of confidence that the conference had achieved its objective.
5. The client also gained data on ideas of how to improve the conference in future years.
What the client said:
“ITD delivered exactly what we wanted of them, in a timely and cost efficient manner.”
[dottedline][/dottedline]
If you would like to discuss how ITD could help your organisation with evaluating your investment please contact us on +44(0) 800 804 8086 or email at info@itd.com
[dottedline][/dottedline]