Earlier this month, Ben Affleck made headlines when he discussed why he believes AI lacks the “taste” that threatens art forms like film and poetry. “AI can write you a very good Elizabethan-sounding imitation verse; it's not going to write Shakespeare for you,” the star actor said while speaking on 2024 CNBC Delivering Alpha.
A new study from University of Pittsburgh researchers begs to differ. The report found that participants were not only unable to distinguish between AI-generated poetry and human-written poetry, but actually preferred those created by AI. and Edouard Macery.
After collecting the work of ten famous poets including William Shakespeare, Walt Whitman, Emily Dickinson and Allen Ginsburg, the researchers used OpenAI's ChatGPT 3.5 to create poems in the “style” of each author. After five AI-generated and five authentic tasks from more than 1,600 participants, they asked them to judge whether the task was created by a human or a large-scale language model (LLM).
The participants were not only able to accurately identify the works produced by the AI, but they were more likely to judge them as human-written than those written by real poets. The five poems with the lowest “human” ratings were all written by humans, while four of the five poems with the highest “human” ratings were created by AI.
Why do AI-generated poems rate so well?
The study also asked more than 600 people to evaluate more than a dozen quality measures from rhythm to originality and evidence. Overall the AI poems were rated very well, although the ratings dropped when participants were told that the poems had been created with AI ahead of time. “Non-poetry readers prefer AI-generated poetry that is accessible, conveys feelings, ideas and themes in direct and easy-to-understand language, but they expect AI-generated poetry to be worse; therefore they mistakenly interpret their liking of the poem as evidence that it was written by a human being,” according to Porter and Macery.
The authors note that the increasingly good translation of poetry produced by AI is a recent phenomenon fueled by the development of LLM. A previous study using poems created with OpenAI's GPT-2, for example, saw participants easily distinguish AI from works written by humans.
The implications of AI development are not limited to poetry. Recent studies have found that AI-generated sketches are increasingly rated higher than human-made ones, while AI jokes are found to be funnier or funnier than human responses. Even the faces produced by the new technology will become more and more difficult to distinguish from those of humans.
Now, the same development can be seen in writing. “These findings represent a leap forward in AI production capabilities,” the study's authors wrote, adding that “poetry has previously been one of the few domains” where AI models have not yet reached such levels of sophistication.
