In 1959 Donald Kirkpatrick introduced four levels of evaluation to the learning profession: reaction, learning, behavior, and results. Thirty years later, Gloria Gery courageously informed us all that we might as well just weigh our students before and after learning, rather than use the metrics we were using then and still using now to determine the effectiveness of what we do for organizations.
To Gloria’s credit, she offered revolutionary insight regarding how we could break free from evaluating learners’ reactions and learning mastery (Levels 1 and 2). She proposed a way for us to validate the transfer of learning to the workplace and measure actual business impact by measuring performance in the flow of work (Levels 3, 4, and beyond).
According to Gloria, this validation of transfer can only happen within a performance support framework empowered by an EPSS, “because the connections between performance support [instances] in the actual work context is so much more direct than the distance between training events and work performance.”
Last year the Performance Support Community completed a year and half long 5 Moments of Need benchmarking effort. One of the areas of focus was measurement. Because a 5 Moments solution extends L&D’s reach into the workflow via an EPSS, it provides unprecedented opportunity for continuous measurement and reporting. We wanted to understand how participating organizations can adjust their measurement strategies for gathering and reporting “business impact” because of this.
One of the primary outcomes of this part of the benchmarking study was the modification and validation of 18 “business impact” outcomes initially proposed by Gery. The following figure shows them categorized into 5 areas.
Figure 1: Business Impact in Five Areas
Gathering measurement data for these outcomes requires a performance support infrastructure with at least one fully functioning EPSS. The EPSS must be properly designed to facilitate this measurement. In addition, the technology behind the EPSS must allow two key functions:
- Targeted usage monitoring
- Spot verification via micro-polling
Targeted Usage Monitoring
A fundamental reality of performance support is that people will choose whether or not they use it. That choice is based upon how helpful the performance support solution is at the moment of need. If it’s helpful, performers will return. If not, they abandon ship quickly and are reluctant if not resistant to returning. So, if a performer, for example, accesses the steps of a specific task multiple times, you can confidently know that those steps are proving helpful. In addition, whenever a performer demonstrates a pattern of accessing a specific resource associated with that task, then you can also confidently assume the resource is proving helpful. By targeting specific tasks and the types of resources associated with those tasks, usage monitoring can provide a valuable ongoing data stream to help determine business impact.Spot Verification via Micro-polling
In the realm of workflow performance, job tasks aren’t all equal when it comes to the consequences of failure. They are all important to the success of the work, but some have significant to catastrophic impact if performers fail to successfully complete them. These “high impact of failure” tasks and their associated resources merit periodic (spot) verification of successful completion as a second data point along with consistent usage. This spot verification can be accomplished via micro-polling. Obviously, this needs to be done purposefully with restraint. But this verification adds credibility to usage data and provides expanded capacity to measure business impact.How Usage and Spot Verification Work Together
One of the five key areas for business impact, shown in figure 1 above, is increasing profitability. Here’s how an organization can go about gathering specific measures on this area via an EPSS.Measuring Reduced Time to Changed Performance:
Suppose a Moment of Change occurs that requires a performer to unlearn and relearn a task using the EPSS for guidance. The beginning of the Moment of Change is time stamped by the occurrence of a change in content in the EPSS. The end time is also stamped when the performer successfully completes the task. Multiple visits to the same task-level support, then, strongly suggests successful performance over time. This can rightly be interpreted as “changed performance” or “proficiency.” You can also include spot checks to verify successful performance by micro-polling performers.Specifically, then, to measure reduced time to changed performance, you would do the following:
- Push notification of the change(s) with a brief description of and direct links to those places in the EPSS that would help performers navigate the changed performance.
- Track usage patterns for all changed tasks and their associated resources that would influence successful performance.
- Confirm via micro-polling successful performance of tasks with changes (especially where the impact of failure is significant to catastrophic) via micro-polling.
Measuring Reduced Time Spent Supporting Others:
When employees are engaged in self-support or peer support, they aren’t performing the work that furthers the mission and success of the organization—and the work for which they are being paid. The purpose of an EPSS is to be a digital coach and provide 2-click/10-second access to just what’s needed, when it’s needed, to successfully perform their work. As such, EPSSs are specifically designed to reduce support costs in this area.The cost of self-support includes the amount of time performers take to solve a problem, figure out how to perform a task, find resources needed to help execute the work, or make a decision. Once an organization has baseline data regarding the amount of time employees spend remembering how to do things, finding resources to do their jobs, and solving problems when things go wrong, it can begin to track EPSS usage patterns. These can be correlated with the baseline data to substantiate reduced self-support costs. Usage patterns to consider are:
- Revisits to a task area and to its specific resources.
- The types of resources accessed.
- The amount of time performers spend in a given task area working with those specific resources.
Measuring Reduction of Work Stoppage to Learn:
Work stoppage costs organizations. One of the fundamental costs of formal learning is that learners, in order to participate, stop working. The time required to adapt what is learned to the workflow also costs. When things go wrong and learners stop work to solve the problem, more productivity is lost, which costs even more.An EPSS allows organizations to safely push, on the average, half of the learning requirements completely into the workflow where performers learn while they perform their work. This reduces work stoppage to learn. In addition, an EPSS can dramatically reduce the time it takes to achieve effective performance in the workflow. And, as mentioned, at the Moment of Solve, an EPSS can reduce work stoppage by providing 2-click/10-second access to just what’s needed to solve the problem. Reduced work stoppage alone can pay for the investment in building and maintaining an EPSS.
These, of course, are just three examples. But targeted usage monitoring and Spot verification via micro-polling can provide similar measurement support for the remaining 15 criteria. There is a measurement strategy for each one. As more organizations mature in their EPSS production capacity, they will be more prepared and able to pursue these and many other measurement capabilities.
This year bodes well for advancing in our capacity to continuously measure business impact. The 5 Moments methodology is being embraced by more and more organizations. We are rapidly reaching a tipping point with organizations ready and capable of measuring performance in the flow of work. If your organization has an EPSS technology infrastructure in place and at least one deployed EPSS, you’re ready to measure performance in the flow of work. Exciting isn't it!
More resources on The 5 Moments of Need.
No comments:
Post a Comment
Note: Only a member of this blog may post a comment.