People Can’t Tell AI From Shakespeare — They Prefer AI’s Verse, Study

“What’s in a name? That which we call a rose, by any other word would smell as sweet.” William Shakespeare wrote this famous phrase for the play Romeo and Juliet to convey the meaninglessness of surnames and labels separating the star crossed lovers. How ironic that nearly 430 years later, AI is seemingly able to remove the meaning and distinction from the writings of The Bard. Let me explain.

Researchers of a study published in Nature Thursday found that participants consistently rated AI-generated poetry more highly than the poetry of well-known human poets based on a variety of qualities. In addition, the study subjects were unable to reliably distinguish between AI-generated poetry and human-authored poetry — even when those humans were some of the most famous names in English writing.

Brian Porter is a postdoctoral research associate and data analyst at the University of Pittsburgh, and he is lead author of the study. He wrote in an email exchange that the primary purposes of the research were to determine if individuals could tell whether a poem had been written by AI or a professional human poet and possible drivers behind the qualitative assessments of the work.

“Poetry is a unique form of text. In some sense it has more rigid structural rules of rhythm, meter and format than other forms of text, and yet in some sense there’s more freedom because you can break any of those rules at any time for artistic reasons,” Porter explained.

“How AI went about imitating that, and how human readers interpreted a poem’s obedience or violation of rules and expectations was an interesting question. Poetry is also generally taken to convey or aim to convey deep truths about the human experience, and it was interesting to test AI’s ability to recreate that,” he added.

Generative AI Created Five Fake Poems In The Style Of The Greats

The study protocol used five poems from 10 famous English poets including Shakespeare, Geoffrey Chaucer, Walt Whitman, Emily Dickinson, Allen Ginsberg and Sylvia Plath. The researchers enlisted ChatGPT 3.5 to then generate five new short poems in the style of each author. The 1,634 participants were assigned one of the 10 authors and a randomized list of the 10 poems for each author. The participants had to pick which versions were genuine or from generative AI.

Then a new group of 696 participants had to do a qualitative assessment on five of the genuine author poems and five of the generative AI poems — all were randomly selected but each author could only have one poem attributed to them. The results in the table below show that AI beat all the human content across the board.

“The most surprising result is how little formal and structural features of the poems seem to matter. Our participants looked almost entirely at the content of the poem when deciding whether or not the poem was AI-generated. This is surprising, and may be part of the reason that the poems were indistinguishable,” Porter wrote.

“A paper just came out recently, which shows that ChatGPT does have specific stylistic tendencies in rhyme, meter, and vocabulary. But if people only look at the content of the poem, those stylistic clues can easily be missed,” he added.

People Like AI Poetry Because It’s Easier to Understand

Porter went on to explain that he and co-researcher Edouard Machery believe that participants preferred AI-generated poems to the classics because the writing, rhythms and rhymes were more accessible.

“You can pretty easily understand what the AI-generated poems are saying on your first read, with limited context. Whereas to appreciate a poem by Chaucer or Plath, you may need to know some context and you may need to spend more time with it,” Porter noted.

But People Say They Prefer Human Writers Over AI

Another interesting finding was that even if participants couldn’t distinguish between AI and human written poetry, once they knew which was which — there was a clear bias and preference toward human authorship.

“Ratings decrease across the board if you tell people that the poem they’re reading was generated by AI. So there’s a risk of deception here: someone could provide AI-generated text or art, pretend that generative AI was not used, and then people might spend money they wouldn’t have been willing to spend if they’d known the truth, because they can’t tell the difference without being told,” wrote Porter.

Despite that risk, he stressed that poets won’t be losing their jobs soon. Porter noted that great poetry often says something that hasn’t been said before, or says something in a new way. If that’s the goal, then AI is at a disadvantage.

“AI doesn’t have experiences of its own, and it’s not trying to communicate ideas or information to you. AI is trying to mimic the patterns of human language production, and it has read everything that’s ever been written. AI does not know love or grief, but it does a great impression of someone who does,” he concluded.