Training Public Sector Employees: Which Employees Gain the Most?
By Frank Mulhern and Jenna Massey, Northwestern University - February 2013
Click here for a downloadable PDF of this research paper.
- Key Evaluating Training Effectiveness
- Explaining Variablility in Training Effectiveness
Public sector employees face particular challenges because of the nature of the work and limitations in opportunities for salary increases and career advancement. In this paper, the Forum explores the extent to which formal training programs can enhance the performance and quality of workplace experience for public sector employees. Our objective is to determine what factors explain variation in the efficacy of training and for which types of employees training was must valuable.
A series of two-day training workshops were administered for employees of a Midwestern state department that offers career development and job search services to unemployed people. The training took place at different regional offices over a period of several months and was provided by a private consulting firm. The training program featured numerous work management skills such as in-person interviewing, note-taking, document management, and using job databases and other online tools.
An online survey was conducted in the spring of 2012 to evaluate the degree to which employees felt the training improved their performance. The survey also evaluated employees’ overall perceptions of the workplace and their experience in it, including perceptions of supervisors and descriptive information on the employees. Just over 900 counselors were asked to complete the survey. A total of 513 completed the survey for a response rate of 56%. The high response rate can be attributed to the fact that regional supervisors sent the requests for participation directly to the counselors in their region.
Between four and ten months passed between the time employees attended the training and when they and were surveyed. The survey was administered at a single point in time. The survey was intentionally administered several months after the training to give employees time to apply the skills they learned in the training and reflect on the impact of the training. For survey questions regarding work management training, the analysis was intended to determine how effective the training was and, in particular, for which employees the training was most, and least, effective.
In the survey, employees were asked to evaluate how their work performance improved after training over twenty specific job functions. The response scale featured five options: 1) no change; 2) slightly improved; 3) moderately improved; 4) substantially improved; 5) dramatically improved. Employees were asked to evaluate the impact of the training for each of the following skill areas:
- Effectively managing time
- Applying active listening technique
- Developing career pathways with customers
- Informational interviewing
- Building stronger relationships with customers
Table 1 shows the level of improvement employees felt they obtained for each of those skill areas. Note that since not all employees received every training module, the number of employees responding to each question is less than the 513 who participated in the survey.
Improvement for Skill Areas
|Evaluation Scale||Effectively managing time||Applying active listening technique||Developing career pathways with customers||Informational interviewing||Building stronger relationships with customers|
|No improvement||127 (35.7%)||86 (24%)||98 (27.3%)||89 (25%)||101 (28.1%)|
|Slightly improved||77 (21.6%)||94 (26.3%)||93 (25.9%)||85 (23.9%)||83 (23.1%)|
|Moderately improved||99 (27.8%)||99 (27.7%)||94 (26.2%)||105 (29.5%)||94 (26.2%)|
|Substantially improved||40 (11.2%)||59 (16.5%)||60 (16.7%)||58 (16.3%)||59 (16.4%)|
|Dramatically improved||13 (3.7%)||20 (5.6%)||14 (3.9%)||19 (5.3%)||22 (6.1%)|
|Total||356 (100%)||358 (100%)||359 (100%)||356 (100%)||359 (100%)|
The table shows fairly consistent levels of improvement across the five skill areas with the exception of “effectively managing time” scoring lower than the other areas. For all skill areas, relatively few employees (about 20-25%) indicated that their skills improved substantially or dramatically. The most prevalent responses were “slightly improved” and “moderately improves” for all skill areas. State administrators and the training company were disappointed that employees did not report that the training had a major effect on the performance.
These results lead one to question why so many of the participants did not think the training did any better than moderately improve their skills. One also is led to wonder what it is about those who indicated substantial or dramatic improvement that makes them different from other employees. To investigate these issues, we developed a single composite variable that represents responses to questions regarding the employee’s level of improvement on twenty specific skills. We call this variable the “overall improvement score” and use it in subsequent analysis.
To explore the impact of the training, we conducted a mean-split analysis of the overall improvement score. Each employee was categorized as being either below or above the mean of this score. We can then compare the two groups to determine what factors relate to an employee being either above or below the mean. We find that several characteristics of employees do not differ between employees in the low versus high effectiveness groups. Among these are gender, age, education level, annual salary, job function or how long they have been working for the organization. We did find several factors different between the two groups. The employees had overall improvement scores above the mean have the following characteristics:
They access the online software more often to help their customers.
Survey participants were asked “In a typical week, how many times do you access the online tool?” The higher scoring group had the mean of 2.45 (on a four-point scale) times accessing the tools each week, while the negative group has the mean of 2.18. This difference indicates that the positive group of employees is more active in using the online software or tools with their work and likely results in them being better counselors.
They initiate more frequent contact with their customers.
Participants were asked “In a typical week, how many times do you initiate communications with each of your customers?” The higher scoring group had the mean of 3.84 (on a four-point scale) weekly outbound communications with each customer, while the negative group has the mean of 3.10. This difference indicates that the positive group of trainees is more active in reaching out their customers.
They are more likely to feel they provide sufficient or excellent service to the customers.
Participants were asked, “Which of the following statements best describes how you feel about the value of the services you provided to your customers?” The two most favorable responses were “sufficient” and “excellent.” The positive group has the mean of 2.51 (on a four-point scale) while the negative group has the mean of 2.23. This difference indicates that the higher scoring employees feel they provide sufficient or excellent service to their customers.
They feel state policies are clear to them.
Participants were asked, “To what extent are the current state policies clear to you?” The group with the higher overall improvement score has the mean of 3.96 (on a five-point scale) while the other group has the mean of 3.66. This difference indicates that the higher scoring group understands state policy better.
They are more motivated to perform well at their work.
Participants were asked, “Which of the following statements best describes how motivated you feel about your job?” The high scoring group has the mean of 1.16 while the negative group has the mean of 1.28 (on a three-point scale with one meaning highly motivated). This difference shows that the higher scoring employees are more motivated to perform well at their work.
We conducted a regression analysis to predict which characteristics of the employee or the job led to higher or lower levels of self-reported improvement based on the training.
Regression Model for Overall Improved Work Performance
Dependent Variable: Overall Improved Work Performance R² : 0.131
|Independent Variable||Standardized Coefficients||t||Sig.|
|Online tool usage||0.13||2.05||0.041|
|Client communication initiation||0.19||3.258||0.001|
The statistical analysis uncovers five variables that influenced the degree to which the employee felt the training led to improved performance:
- Online tool usage – use of job databases and other online sources
- Client communication initiation – number of times an employee initiates contact with clients
- Job satisfaction – the counselors reported level of satisfaction on a five-point scale.
- Job motivation – the counselors reported level of motivation on a three-point scale.
- Workload – the number of clients a counselor is assigned at a given time
The highest coefficients (0.19) are for customer communication initiation and the size of the employee’s workload. The employees who regarded the training as improving their performance are the ones with the highest workloads and those that make the most frequent communications to clients. Three other variables, online tool usage, job satisfaction and job motivation, had significant, but somewhat lower impact on the effectiveness of the training.
A key finding here is that the training worked the best for employees who seem to work the hardest (high workload and number of communications) and are most motivated and satisfied. The take-away from this is that training does not work for employees who are not working hard, motivated or satisfied. Or, put another way, an organization can’t train poor performers to be better if they are not engaged in the workplace.
Training programs are often seen as solutions to poor performance. The results of our analysis of the impact of training on public sector employees show that the effectiveness of training is moderate and variable. Training worked best for employees who were already engaged in the workplace and working hard. It appears that intangible elements such as how employees feel about their jobs are important. The challenge with training programs is that they usually center on specific job tasks and are limited in their ability to influence intangibles such as motivation. Our results show a correlation between motivation and the degree to which employees report that the training improved their performance on specific tasks.