This blog is generated from the Performance Matters Podcast episode titled ROI is Measured in the Workflow. In it, Bob Mosher and Dr. Con Gottfredson are joined by APPLY Synergies’ Executive Director of Consulting Services Sara Chizzo. Together, they explore how embedding learning in the workflow empowers learning professionals to finally measure impact, effectiveness, and return on investment of learning solutions.
Bob Mosher (BM): Welcome back to another Performance
Matters podcast. I can't tell you how excited I am about two things: 1) the measurement
topic, which we really need to talk about after an event I recently attended,
and 2) the people who are with me. They are two of the most remarkable
colleagues that I'm fortunate enough to work with. First, the famous Dr. Conrad
Gottfredson. Welcome back. Good to have you here.
Con Gottfredson (CG): Good to be back.
BM: And this next person just rings our bell. I've
been fortunate to know her professionally and watch her work for about 20 years,
and now we are fortunate to have her as part of our group. I'd like to
introduce you to Sara Chizzo. Sara, welcome to the podcast.
Sara Chizzo (SC): Thank you. It is a pleasure to be
here and a pleasure to work with you both.
BM: It's just a dream come true for us. Sara has a
remarkable history in measurement. Sara, can you tell us a little bit about
your journey?
SC: Absolutely. About 25 years ago, I pivoted to the
learning and development space working for a technical training company called Productivity
Points. That was my first foray into professional learning and understanding
that companies actually pay for this kind of stuff, and for external providers
and experts to help them train and develop their employees. After a few years
of doing that, I got a little bit frustrated. My largest account was Motorola
Solutions. At the end of their contract term, they came back to us and asked
some questions: what did we train on during this period and what was the return
on that investment? First, we couldn't even provide them with accurate
information globally about what we'd actually trained their people to do. The
systems didn't talk to each other. But we really didn't have any way of measuring
the impact, the effectiveness, and the return on that sizeable investment. It
was about that time that I joined a colleague of mine named Kent Barnett, who
started a company called KnowledgeAdvisors with the desire of bringing some
additional discipline to the space. We wanted to help provide more information
and data to companies so that they could understand whether their programs were
moving the dial. So, that was my journey. I’ve spent about 18 years in
measurement and analytics for learning.
BM: Spectacular, and great work. We are really
excited to have you here, because now we move into the workflow. Now we move
into performance. Sara, why don’t you share a couple interesting stats? We're
going to frame this discussion with some things we've heard lately that,
frankly, have been a little troubling. So, why don’t you give us a couple to
start?
SC: I'm first going to totally indict myself and the
work that I did (or didn't) do over the last 18 years with the first two data
points. I think it's important to understand where we are and the data about
measurement in the learning and development space to better understand where we
have been able to move the dial. So, I want to share a couple of things. The
first is a data point from a McKinsey & Company study. Learning leaders say
that only 25% of their programs improve performance. I remember the first time
I read that. I asked myself, “What are we doing?!” What are we doing if, absent
of any information, our learning leaders are instinctively saying that three
quarters of our programs aren't doing anything? What are our stakeholders
paying for? And then the second data point is from a follow-up study that the Performative
team did where they asked learning leaders a bunch of additional questions.
Ninety-seven percent of those polled (I think it was a sample size of about 250)
said that there was waste somewhere in the learning process, but they had no
idea where. Essentially, we were providing learning that wasn't netting an
impact, either to the individual's performance or to that of the organization (talent
or business outcomes). And now that I've had the benefit of looking at this
from the 5 Moments of Need (5 MoN) lens, I cannot help but go back to the
source, which is how we think about supporting our learners at the time of
their work, as they're performing.
If you think about the first data point about three quarters
of programs not improving performance, it's because we're not taking a
performance lens to the development of our programs and the development of our
solutions in the first place. We're not designing for performance. So, if we're
not designing for performance, then we shouldn't be surprised when three
quarters of our programmatic investments don't net a result. And to the second
point about waste in the process, there's far too much of a disconnect between
the actual work that's being done and where we're looking to try and support
learners in doing their work. Of course, there's going to be waste in the process
because the learning is not happening while the work is happening. So, those
two data points really jumped out at me within the context of workflow learning
and the 5 MoN because I think they both could be dramatically and positively
impacted with the right design approach up front.
BM: Perfect. Con, what do you think about when
someone says, “But you know, Con, it's busy. It's hard. There are just so many
influencers out there after the training. My training could get lost in that.
It's not a fair measure of my training, because the learner’s manager, the
learner’s discipline, how soon they try to practice when they get back…all
those things are unfair to measure my training against because those
influencers cloud the measure.” So, what's your answer to that? What do you
think that points to?
CG: Every time I hear that it tells me they're
chasing a skunk down a hole. They're in a training mindset. They're certainly
not thinking, designing, building, and implementing around enabling effective
job performance. Gloria Gery saw this. What a remarkable visionary she was. In
the 90s, when she wrote her book Electronic Performance Support Systems,
she was very clear. In terms of indicting what was being done under the
umbrella of training, she said that it wasn't leading to performance and that
it needed to. And what she saw was that you can't measure impact if you're
trying to get there from the training alone, because too much happens after the
fact. By the time you measure learning, those learners have had to access a lot
of other things to get where they need to be, because the training wasn't
enough. That's the bottom line. The training was not enough to get them to
productive performance. So, what do they do? They rely on other people, they
work through other systems, and they do other kinds of things to achieve
performance. Gloria said that if we build a performance support system that
supports people as they do their work, that system—in the work and the
workplace—gives us the ability to gather data to make those direct connections.
Back to your question, when I hear that, it just lets me know that they're
looking at the training, and the only way learners get to productive
performance is by going and involving other things [besides training].
BM: I’ve shared my frustration around what I heard
recently, which is, “Well, then let's just back off this. I'm sick of chasing
the ROI thing. It's hard.” That's like a fireman saying, “Well, I just don't
want to know if the fire is out because it's hard to sift around and dig in the
rubble. It's hard to get into the dirt and the afterburn because it's messy.”
But until we get to that level and understand that there are embers there, we
don't go beyond just throwing water on the fire. For so long in training, it has
been just that—and we see the flame go out. We assume the learner did well.
They like the experience. They feel like they can apply what they learned and
what they heard was relevant. So, we pick up our trucks and leave, but the
learner is left with the mess of work and the reality of how messy and hard and
volatile the workplace is. So, we have to get beyond training and training
alone. I want to be careful. We're not condemning training as an entity. We're
just saying that, for too long, it's been a safe place and our only answer to this
world we need to journey into a bit more.
Con, talk a bit about this workflow thing and why it's been
such a missing part of our analysis and our understanding for so long. Don't
SMEs give us that? They tell us all that's important when we put them in a room.
CG: They could give us that if we ask the right
questions. We generally go into a task analysis or whatever analysis we're
doing with the mindset that we're going to build a training solution—not that
we're going to enable effective performance on the job. And how are we going to
enable effective job performance without facing the workflow? That's where
performance occurs. Glory Gery called the workflow the “performance zone”. So,
you have to face that; therefore, when you're dealing with effective job
performance, the measurable objectives include the ability to complete a job
task, whatever that task is—and we can measure that, we can gather data around
that—but we've got to be able to face it. And you can't face that without stepping
into the workflow and mapping it.
Let me just say this. Sara, you mentioned waste. When we
look at onboarding programs and we step into the workflow, and we build
workflow solutions that support people as they move through the training, as
they move through that transfer phase, as they move into and begin to sustain
performance in the flow of work—the moments of Apply, Solve, and Change—when we
build that kind of a solution, we consistently see that time to proficiency is cut
in half. I mean, we just saw a client take an 18-month time to proficiency down
to 5 months, because they focused on performance. They brought in the power of
workflow learning and they ended up with people being able to perform more
effectively, more productively, and with less oversight. They were able to
measure and demonstrate all those things because they were facing the workflow
and designing and building and measuring around that.
BM: Sara, as you have learned more about this and you
think about the world you came from, why does a Digital Coach excite you? As
you look at L&D wanting to get to KPIs and other things we've talked about
for so long, what does a Digital Coach add to a measurement conversation? Why do
you think it gives us a different level of impact and approach?
SC: I think it’s really important to remember what
we're in service of as learning professionals within our organizations. We are
in service of the business. We are in service of performance. What gets me
excited in thinking about how we provide solutions that allow our learners to
optimize their work—while they are building their capabilities and their
ability to do their work as effectively as possible—is that it really solves
the measurement problem, right? The measures that we've been focusing on so far
around waste and scrap learning, misalignment—those are almost entirely
assuaged when you actually have a Digital Coach sitting shoulder-to-shoulder
with you in the workflow. Because whatever my work is, I’m going to encounter a
moment when I don't know how to do something. I'm struggling, and if I can
quickly get the answer to my question and the support that I need, then I can
get back to my work. Then we can say the work stoppage was a matter of minutes
vs. other types of measures. For example, 60 or 90 days after a learning event,
we hear from learners that they were only able to use 3% of the training we
provided back on the job. I mean, how is that useful and helpful for a company that
is in a very competitive industry and that’s trying to improve profit margins
and overall competitiveness? So, that's what excites me.
The data points that I shared in the beginning of this
conversation should be a call to action. We need to build our training
differently. We oftentimes translate those data points to mean we need to find
out what's happening with our learners so we can either improve the front-end
training or provide them with some type of a job aid on the back end. Neither
of those things is going to solve the issue. We need to robustly support them
while they're doing their work.
BM: Anyone who can work “assuaged” into the answer to
a question is in a whole new world for me. That was stunning, Sara. Love that.
So, let's talk about something I heard recently. I’d love to
get your reactions. One measure we use is confidence, or self-efficacy, which
is to feel that beyond just remembering everything, I have the metacognitive
skills and tools—it's a combination of both—to enable and improve my
performance. Recently, in a more traditional analysis, a learning professional
learned that practice (aka “doing”) builds confidence, so their solution (to
your point, Sara) was to go back to the front end—in a training mindset—and
build more practice into the class. I agree that practice is better than 50
more PowerPoint slides. I'm all for that because of the cognitive load, etc.
But that's confusing practice with true confidence building. Because even after
I leave, having practiced a lot, I still just don't know. I'm entering the
world of real work.
Con, run at this practice thing for me for just a second.
CG: Well, years ago, we did some work for the world's
largest manufacturer of pumps. It was a European company and when they
approached us, they said, “When our people finish their onboarding training,
they are so confident. They leave, we survey them, and they're very confident. Even
at six months, they are still confident, but something happens at the one-year
mark. They suddenly say their training and onboarding experience was terrible and
they have no real confidence in what we did for them. Can you explain why?” And
I said, “Well, it takes them a year to figure out that what you were doing for
them didn't help them.”
Albert Bandura did the most salient body of research in
terms of self-efficacy and confidence building as it relates to learning. He
found that the sooner people perform effectively and can recover if they make a
mistake, that's the best way to build self-efficacy. Performing effectively in a
controlled classroom is very different than performing effectively in the
workflow. As you said the other day, Bob, the most powerful practice is work. That,
to me, is a profound statement in and of itself. Work is practice: it's
applying, it's doing the work in the workflow. The moment a person successfully
performs on the job, that's when their confidence is reinforced and grows. The
sooner we can enable effective performance in the flow of work, we begin to
build that self-efficacy. The sooner people can recover when they make a
mistake, we build that self-efficacy. Employee engagement is at the heart of self-efficacy,
according to Bandura. That's where we've got to focus. It’s not about more practice
in the classroom. It's about making sure that when people step into the
workflow, they have the help they need from the Digital Coach (aka EPSS) to
perform effectively on the job.
BM: If you want to practice anything, practice your Digital
Coach. I don't want you to leave training because you did 10 practice runs of
the same activity and got to the point where you think you can do it in your
real work. What if I had you do 10 practice runs with a Digital Coach, so that
when you leave, you know where to find what you need (you know how to recover)?
Failure is a remarkable teacher and it's going to happen in the workflow. What
if I mitigate your time to remediate if I help you avoid failure by teaching
the practices around using a Digital Coach while performing so you do things
correctly? That's where performance improves and confidence is raised. So, it's
confidence in my ability to troubleshoot and survive in the workflow vs. confidence
in my ability to memorize well.
SC: I'll give you an example. In between a couple
tours of duty in the learning measurement space, I went to work for one of the
most renowned business schools in the world, which also provides leadership
development to corporations. That was the business unit I worked in. I was
immediately going to be taking over a team and I had some things that I really
needed to address with its members. Talking about this issue of confidence and
coming into a role like that, I remember going through my onboarding process
and being quite stressed about my personal brand and my reputation. I was
asking myself, “Am I going to be able to hang with these people who are pretty
incredible leaders?! Because we do world class leadership development!” I was
worried I might mess up the first performance conversation, or the first time I
had to coach somebody. I would have felt 100% more confident if I had come away
from that onboarding with curated resources in a Digital Coach that were aligned
to the work I was doing. I had experience doing the work I was being asked to
do, but everything's a little bit different company to company and I hadn't
flexed certain muscles in a while. So, I think that’s where we need to think
about the confidence component. Onboarding is a great example of that. We want
folks to feel confident coming out of onboarding, but we want them to feel
confident because they know they're going to be well supported.
BM: And I think we confuse support with training or
learning sometimes, meaning we don't think it's the same. I don't think a
learner looks through the lens of “this is a training asset”, or “this is a support
asset”. They look at it as a performance asset. So, if it helps me perform, and
I learn while doing, that's training (in a way). Right, Con?
CG: Gloria Gery referred to that as unconscious
learning. What she observed is that when you're in the workflow and you're
doing your job, you're learning. If you have a tool to help you do that job,
you are learning. It's not conscious and you’re not in a classroom. Again, she
called it unconscious learning. But let's admit this: no matter how powerful
and wonderful a training class is, when a person leaves that class, they are
not competent. They are not proficient. They're ready to start. They're at the
beginning stage of that, but expertise is developed over time through
experience in the flow of work. So, if you want somebody who has expertise,
it's not going to come from the classroom alone. It's going to come from a
classroom combined with the workflow and experience over time. And that's real
learning. Real learning happens in transfer. Real learning happens in those
real-world practice activities, Bob, that you mentioned, when people are doing
their work.
BM: Let's change the narrative. If we want to measure
performance, let's live at the point of performance. Let's not live only in the
weeks, days, and months before performance and try to correlate. Until we make
this pivot, until we understand the workflow through analysis, until we enable
it with a Digital Coach, until we understand the architecture and design of the
performance support pyramid, criticality, and all the things we talked about in
so many podcasts before, we're never going to get into those higher levels of
Kirkpatrick and Phillips. We can talk about them all we want, but when we go to
the C-suite, those leaders who demand these metrics are going to poke and shoot
holes in them.
The exciting thing I like about what we do and about these
podcasts, about the clients we're blessed to work with and folks like you,
Sara, is that we know that the future in this is now. It's no longer something
to talk about. It's no longer something to walk away from. Yes, it's hard, but
it's doable. Sara, I love what you say sometimes: I think we've made
measurement harder than it really is. I've heard you say that over and over
again. We complicate this, partially because we don't understand the narrative.
It gets simpler when you understand it from this perspective.
SC: One of the things that Con has been schooling me
on that I've been so excited to incorporate into some of our thinking around
measurement is really the partner to performance and productivity, which is the
work stoppage piece. In the example that I gave you earlier when I had a job
change, in the absence of a Digital Coach, what did I do? I made good friends
with the best sales manager in that entire organization. Every time I needed
something, I called him, I emailed him, I texted him, I Slacked him, etc. and
got my support through his coaching. I mean, that is incredibly costly, not
only in terms of my own work stoppage, but I was also causing his work
stoppage. Ultimately, I ended up getting a lot of the answers that I needed,
but at what cost? We can amplify performance and we can increase productivity
dramatically without having that work stoppage. That's the important piece that
I think is really the partner to productivity in the ROI conversation that I'm
excited to have now.
CG: Across the board, the cost of stopping work to
learn doubles the cost of learning per employee. It just doubles that cost. And
it's a real cost because people are stopping work. Our goal is to enable and
sustain measurable, effective job performance in a way that minimizes interruption
of the work that employees are hired to do. That requires us to step into the
workflow and support people in the workflow, which at the same time enables us
to measure what is happening in that workflow and directly demonstrate that
what we're doing is making a difference in terms of people being able to
perform effectively.
BM: You know, I don't think we can go any further
than that. That summed it up perfectly. And this is why I'm finally excited
about this conversation. It's been the elephant in the room for my 40 years of
doing this. In the last 10 or 20 years, Con, working with you and getting into
the workflow the way we can now, the narrative changes. We can do this, but we
as an industry must choose to change our deliverable, our approach, and the
conversation with the business—and step up to wanting to do ROI. We can’t give
up on it.
Thank you so much. You're both spectacular. Great podcast.
Great conversations.
Subscribe to The Performance Matters Podcast to stay
up to date on all the latest conversations and guests in the 5 Moments space.
Visit our website for additional resources:
Certificate courses, an eBook on workflow learning, and our latest EnABLE
Methodology white paper.
Join the conversation on LinkedIn.
Be part
of the Workflow Learning & Performance Alliance.
Copyright © 2023 by APPLY Synergies, LLC
All Rights Reserved.
.png)