Forum: Author Hangout

I didn't see that one - thanks for posting.

The article did include a link that actually might be of more interest to writers here.

I clicked it, and then took a look at the study the Guardian article is referencing: https://www.cambridge.org/core/journals/judgment-and-decision-making/article/bot-or-not-can-people-tell-the-difference-between-stories-written-by-a-human-or-by-an-ai-system/45E6DC0BB90AA648654D5AE243F6C667

That study kinda annoys me.

In short: They took ~1.700 people and gave them ONE short-story to read, creating four groups:
1. Read a human-written story they were told is human-written
2. Read a human-written story they were told is AI-Generated
3. Read an AI-generated story they were told is human-written
4. Read an AI-generated story they were told is AI-Generated
Then they were all given a questionnaire to determine whether they paid attention while reading, and a second questionnaire to determine whether they can remember core parts of the story, before they were asked whether they think it was AI-written and how much they liked it.

Now, the guy conducting the study saw people claim the AI-Written story is better, unless they were told it's AI-written, so he suggests AI isn't actually bad at writing, but humans have a bias. But here's my problem with the whole thing:

First, the "story" they were reading was ~1,000 words long. There already are studies that show AI gets harder to detect the shorter a text is. And I find 1,000-word stories to not be... well... stories.
Second, about 15% of the participants failed the attention and memory tests after reading the story. FIFTEEN percent couldn't remember core events in the stories that were merely 1,000 words long. But there's a real chance the true number of people not having properly read the stories is WAY higher, because the questionnaires were multiple-choice, the false plot-points were VERY obvious (like "She yells obscenities and punches a nurse", which is something you should remember if it happened), and you had to claim to have seen at least two false plot-points to fail the test. That should tell you about how "engaging" and "understandable" the stories truly were.
And lastly... the worst thing the University essentially put the debunking of their study's validity into the Summary themselves, but didn't seem to realize that. "Self-reported expertise with AI, but not with fictional literature, was positively correlated with correct story identification."
What they're saying, in essence, is: The less you read, the more likely you are to think AI-generated texts are human-written.

"Self-reported expertise with AI, but not with fictional literature, was positively correlated with correct story identification."
What they're saying, in essence, is: The less you read, the more likely you are to think AI-generated texts are human-written.

Apparently the study was the masters thesis of the lead author, Sydney Sears Master of Science in Psychology from from Villanova University , and the other author, Dr. Deena Weisberg PhD Psychology from Yale, was her advisor.
While some people will try to use the results to extol the virtues of AI the study's focus was human perception and decision making and was thus limited in scope and research as you point out and the authors acknowledge per your quote.
It is a shame that people with a vested interest in AI will use this sort of study to mislead their target audience for personal gain.
How this ended up at Cambridge University Press I couldn't hazard a guess.

Gary

I just checked, but, throughout the study, I also couldn't find any metric for what constitutes "expertise" in their opinion. Am I an AI-expert because I have a ChatGPT subscription, or do I need to be able to build my own transformer?

I think they literally just had the participants check a box saying, "Would you consider yourself experienced in AI/fictional literature?", going by their other procedures...