How good is AI at literary criticism?

it is starting to improve opinion into knowledge.

Until recently, although I was excited about LLMs, I thought they were pretty hopeless literary critics. (And the poetry they write was—and is—the purest dross.)¹ While it was doing well on maths and law exams, it made little progress on English exams.

LLMs have long been good as a “footnote on demand” service. You can drop a confounding line of poetry or a page of Ulysses in and get a pretty good explanation. It will also provide context. Whenever you are stuck wanting to know how to play the card games in Jane Austen, or what “Ineluctable modality of the visible” means, ChatGPT or Claude can tell you.

You still have to think for yourself. You still need to puzzle out the words, think through the fog of uncertainty. But now you can ask for etymologies, test your opinions, and so on. I don’t do this very much, but it’s one way that LLMs have been performing well. (LLMs are also useful for scholarship, such as testing the claims made by scholars at the macro level, but that’s a distinct topic, so I won’t cover it here. This piece is Aeon covers such interesting subjects as how Propp’s folktale theory was updated using AI.)²

But for bigger critical questions, they were dreadful. The main problem was that they seemed to have learned how to analyse literature from those people who think the study of English is a largely subjective affair. You know the sort. They talk a lot about the human condition and they say things like “literature can only teach us about itself” or “literature shows us what it means to be fully human.” Obvious nonsense dressed in the robes of respectability.

What on earth does it mean to be fully human? No-one can say. One reason why literature struggles to attract people in modern culture is that these statements are unbearably vague. Spend thousands of hours reading Shakespeare and Tolstoy and you too could speak in the most mundane platitudes! Give years of your life to the close reading of the great poets in order to wave your hands and talk about the human condition!

I won’t link to any of this work, partly because it would be mean-spirited, but mostly because I don’t want to encourage you to read that stuff.

To begin with, the models spoke like this too. They must have learned it online.³ All that vague and unknowledgeable talk about feelings and impressions meant it sounded like a school child trying to find something to fill up half a page.⁴

That has now changed.

The question I like to use as a test is taken from a 1970s A-level exam.

“Whatever else its merits, The Clerk’s Tale does not achieve its effects through surprise.” Discuss.

The Clerk’s Tale is a test narrative.⁵ To give a good answer, you must deal with the genre, the audience expectations, and preferably with the Bible. You need to explain that it is not real. It is more like allegory, in the vein of the Book of Job. And you need to make it clear that Chaucer’s audience knew that. Knowing what to expect, of course, doesn’t make it less astonishing, even if it makes it less surprising. (I call this the expectation of astonishment.)

In 2023, ChatGPT’s answer was fine. It said things like “The Clerk's Tale is a story that achieves its effects primarily through the use of characterization, symbolism, and themes.”⁶ Nothing really about the genre, the Bible, and so on. GPT4o mini did better, talking about Boccaccio.⁷ But it still talked a lot of flannel: “its primary effects are achieved through its exploration of profound themes, rich character development, and the moral questions it raises.”

Now, o1 gives a better answer yet, starting with Petrarch, noting that the tale is more about character than plot, points out that Walter says he will test Griselda, makes a Christian parallel, and so on. The conclusion is much less vague.

…the real drama emerges from the audience’s foreknowledge and the tension between moral idealism and human compassion.

I also asked o1 about The Wife of Bath, Sonnet 27, Prospero, and the origins of the novel. Each answer showed the same thing. Much more knowledge, and much less of the vague claptrap that passes the wrong sort of English Literature exams. This is far from perfect, but it is improving. The model is learning not to talk about “profound themes”.

Good criticism is specific. Increasingly, ChatGPT is improving at this sort of criticism. It failed at the question of whether the Baroness in The Sound of Music is a Nazi. Ambiguity bothers it. It isn’t very good at discussing mood. But it is pulling away from the genre of criticism that waffles about the human condition. Its answers are less and less torpid.

Sure, it lacks style. And it keeps to the surface. LLMs aren’t going to be writing in the London Review of Books in the short term. But as they get smarter, they will hew more and more to the true purpose of criticism, which is, in the words of Samuel Johnson, to improve opinion into knowledge. And as they do so, they will become increasingly useful to all good common readers, who are themselves in the pursuit of knowledge.

Francesco di Stefano Pesellino - Episode from the Story of Griselda

1. My one success was asking a model to read an Emily Dickinson poem, analyse it, write a commentary, tell me what it had learned, and then asking it to write an imitation. The result was largely poor, but it had one image, describing a postbox during the American civil war as a sentinel of hope that I still remember.

2. “Conversation is one area where computational methodology has been shown to trump the claims of literary scholars – even scientifically inclined ones. In his Atlas of the European Novel (1999), Moretti suggested that the bustling urban setting of much 19th-century fiction tends to involve more characters but less dialogue, compared with narratives set within the confines of the family in the village or the countryside. A group of computational linguists and literary scholars at Columbia University decided to investigate this claim, using software that built a conversational social network from a corpus of 60 novels from the 19th century.

“The software parsed each sentence in terms of its syntax, and then found references to people. It also flagged stretches of quoted speech and attributed the quotes to speakers. This allowed the system to discern who was talking to whom. Although Moretti’s theory predicted an inverse correlation between the amount of dialogue and the number of characters, these scholars found no such statistically significant effect. Instead, they discovered that narrative voice, such as first- or third-person narration, was more relevant than the setting in urban or rural environments.”

3. For example, an LLM told me recently that the problem plays and the romances in Shakespeare are often the same. I haven’t heard this. The only place I could find that information were places like SparkNotes or similar.

4. One problem is that it is possible to verify a wrong answer in the sciences quite easily. Whereas in literature we are more likely to say an answer is bad than wrong, opening up the possibility of mediocrity defending itself with the weak sword of personal opinion.

5. Walter, the Marquis, marries Griselda, a village girl. He decides to test her wifeliness and has her children taken away at birth. She thinks they have gone to die. Another Marquis, however, raises them in secret. Years later, Walter tests Griselda again: he sends her home and tells her he is marrying a younger woman. Griselda dutifully goes home but comes back to help with the wedding preparations. The big surprise is that the children are coming back. Instead of marrying one of them, Walter presents the young woman and the young man to Griselda as her returned children.

6. Here’s another extract from the answer.

The primary theme of the tale is the unwavering loyalty and obedience of Griselda to her husband, Walter, despite the harsh tests he puts her through. Chaucer’s use of characterization highlights this theme through Griselda’s humility and virtue, as she willingly submits to her husband’s will. The symbolism of the name “Griselda,” which means "gray" or "humble," reinforces this theme, as Griselda's humility is a key aspect of her character.

https://chatgpt.com/share/678434f3-9d8c-800b-9783-3a389ed129d0

Chaucer’s audience would likely have been familiar with the Griselda story, which originated in Giovanni Boccaccio’s Decameron and was widely circulated in medieval Europe. This familiarity reduces the potential for surprise, as the audience would anticipate the plot's progression and focus instead on the moral and ethical implications presented. The tale's effectiveness, therefore, hinges not on unforeseen developments but on the contemplation of its central themes and the emotional resonance of Griselda's trials.

7. https://chatgpt.com/share/678436bb-f948-800b-af01-df9e7d10e208