Many teachers who have stared down a tall stack of ungraded papers have wished that they could magically whisk the whole stack, Mary-Poppins-style, into graded papers.
Of course, what’s more important than simply scoring the assignments is giving students feedback that will improve their writing. But with all the demands on teachers’ time, they often struggle to read and offer feedback on such high volumes of work.
Generative AI offers a possible solution—a way for students to put in the hours of writing practice while receiving the real-time feedback they need to improve.
In a recent meta-analysis, researchers analyzed data from 20 studies of automated writing evaluation (AWE) systems—AI-powered tools designed to score and grade student writing. Collectively, these systems provided individualized, process-oriented writing feedback to 2,800 high school and college students. The researchers sought to determine their effects on students’ subsequent writing revisions or performance on transfer tasks.
Overall, they found that AWE systems had a medium effect size (equivalent to a gain of about 7 percentile points on a standardized test), with larger effects for multilingual learners. However, effects varied widely, which prompted the researchers to caution that “automated feedback should be combined with other forms of support, such as teacher feedback and individualized learning opportunities, to ensure its effectiveness.”
Another recent study analyzed the feedback given to more than 130,000 Utah students by an AWE called Utah Compose. This study illustrates the benefits and limitations of AI scoring and feedback on writing. The system scored student writing on a five-point scale using the Six Trait Model of writing (development of ideas, organization, style, sentence fluency, word choice, and conventions) in three genres (narrative, argumentative, and informative/explanatory) across grades 4–10. It gave students immediate feedback (“Your word choices make your narrative strong”) and pointers for improvement (“Try adding strong verbs, specific nouns, adjectives and adverbs”).
During the first year, students demonstrated small, yet positive gains in their writing performance, which tapered off during subsequent years—resulting in gains of 9.14, 4.38, and 1.37, respectively, on the 900-point state assessment of reading and writing. Researchers speculated this plateau might have reflected declining use of the system as students transitioned to high school.
Yet it may also reflect the caution drawn from the first meta-analysis mentioned—namely, that automated scoring and feedback systems should supplement, not supplant, real teacher feedback and instruction. So, while automated systems can remove some of the drudgery of grading, they shouldn’t replace true feedback. Students need to know a real person is reading and responding to their writing.
