Back in January, my friend Bror Saxberg, chief learning officer of Kaplan, published an eye-popping blog about a meta-analysis that Kurt VanLehn published recently about nearly 100 well-constructed papers about computers used to tutor learners.
A couple of headlines from the meta-analysis are worth spotlighting here.
First, the work shines some questions on Benjamin Bloom’s analysis from a couple decades ago that suggested that well-designed human tutoring could deliver around a whopping 2 standard deviations worth of learning performance. VanLehn’s paper suggests that the effect size seems to be more around 0.79 than 2 standard deviations—still, nothing at which to scoff.
Second, as Saxberg details, VanLehn does some important work in splitting up the types of tutoring research by “grain size”: answer-based tutors, step-based tutors, substep-based tutors, and human tutoring, as well as by the type of student behavior, which ranges from passive to active to constructive and finally interactive. Stunningly, the typical answer-based tutoring systems average an effect size of around 0.35 standard deviations, and all three of the step-based, substep-based, and human tutoring cluster around an effect size of 0.75 standard deviations. In other words, some machine-based tutoring is approaching the effect size of real human tutoring—and there is less variation than one might expect as the grain size of tutoring becomes finer. This finding is a startling observation.
Saxberg makes some great points on the cautions and potential of this research, as well as the questions we should be continuing to ask. I just want to talk briefly about this from the angle of disruptive innovation and think about how we might implement these tutoring solutions at scale.
As Saxberg writes, great human tutoring is wonderful if you can get it, but simply isn’t practical at scale. We know that the vast majority of learners that could benefit from tutoring simply don’t have access to any at all (some have suggested this number approaches 80 percent of students). This means that there is a lot of nonconsumption in the tutoring space to launch disruptive innovations that utilize the power of machine tutoring at a much lower price point in a manner far more accessible and convenient than are human tutors to millions or even billions around the world. The wrong tactic for entrepreneurs debuting these solutions is to compete head on against existing solutions where the performance won’t be as good. They should instead focus on where the advantages of convenience, accessibility, simplicity, and affordability are valued and more important than absolute efficacy.
By competing against nonconsumption where this is the case, for those who suggest that the machine-based tutoring isn’t as good as the best that’s out there, that will be the answer to the wrong question, as it will be way better than the alternative—nothing at all. And as the research illustrates, it’s a good deal better than that even at this point.
What’s predictable about technology is that it improves constantly year over year, so what at one point isn’t good enough for most, over time will actually overshoot what many need from it. The lessons from disruptive innovation suggest that these technologies may never be as good as the absolute best human tutor (for example, the raw capacity of vacuum tubes still outpaces that of transistors, which disrupted the vacuum tubes in the consumer electronics market), but they will be plenty close. And as they improve, the machine-based tutoring technologies will become good enough for those who could or would have paid full price or changed up their schedule to connect with a human tutor, such that machine-based tutoring may well be the norm for many of us as a prime mode of learning in the future.
In some ways, the Khan Academy and other start-ups are packing in some elements of these machine-based tutoring systems as they evolve and grow, such that this revolution is really already under way.
When I was writing Disrupting Class: How Disruptive Innovation Will Change the Way the World Learns, I often wondered whether the subtitle should be “For every child, a tutor.” As the research shows, that vision may not be so far-fetched.
