Forum: Story Discussion and Feedback

There should be serious consideration places by the Web God here to mandate all AI stories be tagged so, or they risk being deleted.

He's already using AI detection tools and tagging stories that detect as (I believe he said over 50%) AI generated.

At least one author very publicly took his ball and went home over a disputed AI generated tag.

I strongly suspect that 'author' on the forum was an AI chatbot.

That's why I refuse to fat shame authors I spot using AI.

I would also suggest that there should be some humility as to your ability to correctly identify what is AI.

So far I haven't been proved wrong in incorrectly identifying AI-assisted

There's no good way, short of author admission, to prove it either way. You aren't getting agreement from me here.

Doesn't your opinion that I need your agreement show a lack of humility?

I didn't say you need it. But the way your reply tries to justify your judgements implies that you want it. If you were uninterested in my opinion, why the effort to justify your judgement to me?

but I am curious as to why you think my judgement might be suspect.

Because, actually measuring the accuracy objectively is impossible.

Determining the error rate, false positive, or false negative, is impossible.

You think your judgment is accurate, and maybe it is, but you have absolutely no way of knowing for sure.

From aroslav's recent blog post, I'm considerably underestimating the amount of AI content. The recent flood of reincarnation stories are allegedly AI.

I wasn't suggesting that there aren't any AI stories (or even that there aren't a lot of them).

However, short of a poster admitting it, there is no way to know with certainty if a particular story is AI. It is at best a probabilistic assessment and there is no way to objectively validate the accuracy of those assessments.

I'm not suggesting that you can't or shouldn't say that you think that a particular story is AI.

What I am suggesting is a bit of caution as to the certainty with which those statements are made.

Actually it is possible to measure the error rate of these detectors (and thus also humans, although for humans it's slightly harder, as these learn, so you need much more "test data" or regularly fresh test data).

You cannot determinate if a text is human written or AI generated after the fact.

But you can determinate if you (the researcher) are involved in collecting the text.

So you get your data set.

Then you collect your "white mice" (test subjects, couldn't refuse that inside joke), be it AI detectors, open source or commercial ones, or humans, and run experiments with them.

Then you do statistics on the results.

Then you see rather clearly how well it works.

Reality is (I haven't checked how humans fare, but considering that we lose currently even for videos, shrug), but the state of the art for automatic LLM generated text detection is:

- small number of LLM with standard parameters -> 99% accuracy, with perhaps 5% false positives in the worst cases.

- allow arbitrary LLM or fiddling with the parameters -> and the detectors start to perform little better than random number generators. Basically it's an unsolved problem.

As mentioned, that's not some random opinion, that's the academic state of the art.

Now humans are on one hand a bit better: We notice if some stuff is inconsistent. Which is one of the weaknesses of LLM, especially many of them have limited context, and start to generate inconsistent details.

Now, I've been a book rat my whole life, I've got news here for you, sadly human authors (without the help of professional editors, and even then) do make these continuation errors too. Some are better at it, some are worse, but just because somebody mixes up the names of some auxiliary figure in a story does not make it an AI
generated story, especially not on an online website.

But you are definitely wrong, determinating the error rates is possible, and scientists are doing it all the time, that's literally their daily job.

As mentioned, that's not some random opinion, that's the academic state of the art.

Can you cite an actual research paper on that?

Because I doubt the validity of the claimed accuracy even with a limited number of LLMs and default parameters.

It would be interesting to see the methodology.

Dom

https://aclanthology.org/2025.findings-emnlp.609/

https://www.semanticscholar.org/paper/Automatic-Detection-of-LLM-Generated-Code%3A-A-Case-Rahman-Khatoonabadi/636bd4fe4675949eff7040a8ee35016356f0c9af

https://arxiv.org/abs/2506.23517

That's why I doubt claims that anybody can decide if a text is written by an AI or a human reliably, post facto.

It's the medium (electronic text, text/plain), which as such is by default free of forensics, that leads basically to statistical methods for author analysis.

And these lead to values that are prefixed by something with a lowercase p, if you get the insider joke.

Exactly, there is no way, provably.

An LLM can recreate exactly the same content independently, a human can recreate the same content independently.

That's why the IP mafia inventeted patents.

With physical art, there is at least a physical object you can investigate to check if it's consistent with the claimed history.

With digital text files there are literally no fingerprints, no chemical residue, no paper to analyze, …

Not much to go on, beyond textual statistics, and these detectors as the academic papers I've looked up today have inacceptable false-positives in the best cases, and are a joke when one starts to fiddle with LLM parameters, or instead of using a tiny list of LLMs to generate the "AI texts", all LLM available are allowed to play. Suddenly the "99% accuracy" detectors become not really better then rolling dice. Or as the English idiom is, snake oil.

How do you prove that? That's literally not provable.
Even if the author had a video record of him/herself typing the whole story, you cannot be sure that it was not generated before by AI, right?

It's literally hard for physical art in many cases to decide if a piece is truly done by the claimed original artist or fake.

But for a collection of bytes, by definition, you cannot say if string of bytes was created by careful composition by a human or by a mathematical function.

Yes you can observe in the first place, so you can put labels on files, oh, these were typed in by humans, these were generated by LLM, and derive from that statistical predictors again, but these will stay predictors.

There is also nothing that keeps an LLM from recreating a certain value of a file, nor a human of reentering the file content.

So your claim that you haven't been proven wrong in incorrectly identifying AI-assisted stories, would require a rather complicated controlled experiment, and I doubt that's what you are referring to, right?

How many death-row inmates in the USA were exonerated after they were sentenced upon a verdict beyond reasonable doubt?

As mentioned the threshold needs to be set very carefully (or not, shrug, in classroom environments, the goal can be often arbitrary, if the something else is the training goal), and matched to process and the goals one wants to achieve.

Not sure what Lazeez wants to achieve with running an LLM generated detector on stories.

Anyway, there is a reason why EU law forbids algorithmic output to have direct impact on humans without a real human in the loop.

E.g. as a thought experiment, Amazon introduced an "AI generated" tag for kindle books.

The false positives could have literally livelihood destroying side-effects on innocent authors.

No idea how Canada handles that stuff, nor is storiesonline big enough to be on the radar.

But notice that WLPC also operates ZBookStore so the ruining the reputation of a SO author who is selling books here might also have monetary damages, sigh.

So I certainly wonder what exactly is the idea behind automatically labeling stories as AI content ->

- it's unreliable
- it does not provide value to the users

I mean user trainable private categories (it might be for starters even just Bayes based ones), that would be a value proposition. (I personally have not implemented that for my private use, as the download limit would make it a bit problematic, but in the past year I lost my shyness to doing browser addons, sigh)

Few of those were either factually innocent, as opposed to legally not guilty. Many had not exhausted the standard appeal process.
Time had passed in most of these cases or political pressure blocked retrials with the trial errors corrected.

I've done a nice test on the current state of video generating AIs (they are currently resource limited to short clips), and I literally managed to to detect humans/AI generated clips with around 66% reliability (the clips were short clips, where the first full frame was taken, and the gen AI was asked to create a clip from that)

And that's basically the conclusion on LLM detection too:

However, widely reported "99% accuracy" claims collapse to 19-52% accuracy when tested on unknown models using random sampling with repetition penalties—settings increasingly common in modern LLM deployments.

So basically, the things works against ChatGPT with the default parameters. It still works if you include half a dozen mainstream bots like Gemini and CoPilot to the possible culprits. The moment people start playing with the sliders (which "for creative writing" as a bot, or in an agent style use is basically mandatory), all these "detectors" become snake oil. Especially if you run them in a unsupervised way. If you want a plausible verdict, hire a data scientist that looks at the text and runs a a number of detectors on the text and analyzes the results. That might be possible for an important legal dispute, but not a story website, I admit.

https://arxiv.org/pdf/2410.23746

The irony here is, these "AI written text detectors" become exactly what one is supposed to avoid, "badly deployed, unsupervised AI deployments, where AI output, the verdict is it AI generated or not, is shown to the public without a check by a human".

I literally managed to to detect humans/AI generated clips with around 66% reliability

I question the validity of the assessment.

Assuming for the sake of argument that the assessment is valid, I would call that guessing.

Well, some people in my Mastodon time line with, how did they put it, a master degree with something video/media in the title, managed to have a correct detection rate of 80%.

And I completely believe them that they do know the field a bit, because their explanation of the details that they did check during the 5-10 seconds playbacks of the videos included many things technical aspects I never ever heard off.

Or in other cases knowledge that some effects never happen in such situations, with real physics and optics, which the AI obviously haven't yet heard of.

But for a lay person like me, how the f%ck should I know that this effects don't happen on regular cameras?

I agree on the difficulty finding stories to engage me lately. I am hoping that its just a lag while some the better authors work on new postings.

Honestly, I would rather read a "Mad Libs" story than most of the AI crap I see.

Isn't that what AI stories are? Give a few words and AI guesses the rest.

40+ tags on a 4kb story is nature's way of saying do not click.
ASSTR went down, AO3s terms of service make clear they are only there for data mining. Amazon AFAIK no longer allows incest stories. I guess that particular brand of pervert needed a new home.

Typing a random location, item or idea into the search bar will often turn up unexpected gems. Quite often I've tried to fill a lost story request by searching a relevant line, only to spend the next two hours reading a different story.

Of course he doesn't have anything to back it up, he's just making noise because his bridge is lonely.

Your first example was likely made with Suno. You can see another example here: https://suno.com/hook/8c5f39fb-a96a-4a5d-9433-40075a273615

This is making the rounds on X right now. We're steps away from genAI being indistinguishable from real life.

https://x.com/MAGACult2/status/2008922663000142282

I was really hoping that was going to be a link to the Irish version of Me Mum Died in the Holly.

You do realize that you can split these ~500 token outputs to get smaller chapters, and you can paste multiple chapters into one file to get a longer file.

Plus that seems only for the simplistic workflow of generating stuff with a trivial chatbot, not using more complex tools.

You heard the concept of false positives?

Curious are you willing to tell what you use for as a predictor function for "AI generated"?

Curious are you willing to tell what you use for as a predictor function for "AI generated"?

I hope he doesn't - otherwise people will adjust the AI output to hide. See, for example, earlier threads where spaces around m-dashes was discussed. If people want to send predictors to Lazeez, it should be direct, not via a forum post!

Ah, I wondered, only because I blundered today about an author (who has been here for more than a decade) complaining about his style suddenly triggering false-positives.

And as far as I know the problem of deciding if something is AI written is scientifically not solved, nor completely solvable.

At best you can get a probability score, cynically, that's "AI", if you want, it's probably a ML model that is used to spit out that probability when presented with the input tokens.

Purely philosophically, and from a system design point, it's always a question how this embedded in the whole system and processes.

E.g. take "AIgenerated?" in the context of education.
Checking homework/theses for AI output can be handled in two different ways:

- if AI content is detected, you kick out the student. (okay, perhaps hyperbole, but that's the bad way)

- if AI content is detected, you quietly add the student and the work to the "random" intense oral examination heap. If the student did the work, oral exam will in most cases be no real hardship. If the student basically copied the solution from "fellow student Mr. AI", generally that will show in an exam interrogation.

As I've been preaching for some time now (that AI study course makes one think about these stuff), it's always about the complete system. The same LLM can work quite differently depending upon the "business processes", the software that drives it, the prompts, and other subtle details.

Last but not least, if somebody wants to figure out what predictor Lazeez is using, it's probably a short exercise:
- he mentioned it costs to use it, so that suggests he uses a 3rd party one. Probably not something that he can run locally. Probably also not something that he self-hosts in a cloud, although that's a possibility.

- so you take a sample of say 30 stories, 15 labelled AI, 15 not labelled so, on the shorter perhaps, take a short market overview what AI text predictors are available for Canadian companies, and run these 30 stories through these predictors.

- my data scientist side suggests that you will probably have either a more or less clear favorite for the predictor he uses, OR you'll discover that there is a number of predictors that perform very similar.

- And finally, you can almost certainly, even with the download limits, get a big enough training set from SO directly to train a predictor yourself.

- if AI content is detected, you kick out the student. (okay, perhaps hyperbole, but that's the bad way)

- if AI content is detected, you quietly add the student and the work to the "random" intense oral examination heap. If the student did the work, oral exam will in most cases be no real hardship. If the student basically copied the solution from "fellow student Mr. AI", generally that will show in an exam interrogation.

Both of these "solutions" are problematic at the K-12 level. And yes, there are K-12 teachers in the US complaining about students using AI to do homework.

The first is impossible for any public school. Government schools can't expel students.

The second is impractical. K-12 schools are not set up to conduct routine oral examinations.

I remember reading about a Masters or Phd candidate who was expelled when a professor recognized copying from his own thesis done years back at a different university. The student later admitted that he took some of the thesis from AI as "research." The AI had obviously been fed the professor's thesis. The stated reason for expulsion was, presenting someone else's work as his own.

a Masters or Phd candidate

Yes, at a university. That has less than nothing to do with my comment which was about the impracticality of the proposed solutions at the K-12 (grade school and high school) level.

I don't advocate expelling students, that was given as an example of stupid, bad policy.

And if they are not set up to verify the competence of pupils one on one, then you have an issue with your education system.

I completely understand that this would be expensive and work intensive. Oops.

But then I can compare here in Europe the university of my youth in the 1990s, where for my technical degree practically all exams were "bring whatever you can carry yourself into the lecture hall" open book, and required true understanding of the subject matter.

And then we switched to the bachelor/master system, university suddenly got much less money, the 5 year master-level degree was turned into 8 different specialized bachelors and 4 follow up masters, the catalogue of >100 specialization classes was cut by 50% and turned into mandatory classes in one of the new study courses and electives in the others.

And of course because the number of TAs was curtailed, the exams turned from open book, understanding based to multiple choice, simplified checking if you have looked at the lecture notes.

Funny thing, I'm doing another technical/mathematical study course now that I'm a bit older, and the experience is totally different.

The real odd thing is, that for many classes going through the slides (not lecture notes) 1-2 days before the "exam" is enough to get a passable grade nowadays (at least for me), which is sad, considering that it's still a hard study course with ~80% dropout rate.

So yes, the issue is if your education is set up for "mass production" without the teacher being able to know their pupils, you have a problem, because then the pupils can get away with submitting the output of others (even the output of a LLM in some cases) as their own.

That's actually why in my country our "mandatory schools" (the ones where everyone has to go to, basically the weaker pupils) tend to be set up to have 2 teachers in parallel for languages and math, interestingly the "better" schools in the track for going to university (which can select their pupils, and also eject them) don't have such luxuries, they are harsher.

I don't consider a person that uploads "AI Generated" stories an author but that's just my opinion. I've finished a grand total of 1 story to date and I don't consider myself an author either...

SunSeeker

The pain people are willing to accept.

Chat interfaces (like chatgpt) are absolutely, IMHO unusable for creating/editing/improving things.

But listening here, that seems to be the common modus operandi.

Chat interfaces (like chatgpt) are absolutely, IMHO unusable for creating/editing/improving things.

But listening here, that seems to be the common modus operandi.

I agree with you, general purpose chatbots are unusable for this.
There are purpose built story writing AI engines out there.
I haven't posted anything AI generated yet, but I've been playing around with one (https://novelai.net). The problem people will run into with these is that they aren't free to use.

I hope that's to pay royalties to the authors whose in-copyright works they used for training ;-)

You have got to be kidding. You know it is only to enrich the AI companies.

Well, I personally am interested in creating the tools, and as it happens I do have the hardware at home to run relatively big LLM.

And some of the online gambling sports books are being run with AI's.

And some of the online gambling sports books are being run with AI's.

A lot depends on what they are using AI for.

A lot depends on what they are using AI for.

I wonder what the score was between West Ham and Maccabi Tel Aviv :-)

AJ

From Facebook's Meta AI this morning:

"The Duke Blue Devils lost to the Indiana Hoosiers in the CFP National Championship game with a score of 27-21. However, I suspect you're referring to their recent games. Here are some recent results ¹:
- _Duke Blue Devils vs. Indiana Hoosiers_: The Hoosiers won 27-21 in the CFP National Championship game on January 19, 2026."

That AI summary might lead you to think that Indiana beat Duke in 2026, but the 2026 championship was Indiana vs Miami.

https://www.ncaa.com/live-updates/football/fbs/indiana-defeats-miami-win-college-football-playoff-national-championship-game

I know. I watched the game.
Sad that the bowl committee took the ACC's 3rd place finisher instead of the champion.

"The Duke Blue Devils lost to the Indiana Hoosiers in the CFP National Championship game with a score of 27-21. However, I suspect you're referring to their recent games. Here are some recent results ¹:
- _Duke Blue Devils vs. Indiana Hoosiers_: The Hoosiers won 27-21 in the CFP National Championship game on January 19, 2026."

And that's what a proper AI research agent like perplexity answers, when you paste the above paragraph, and add "Verify these claims for me please, and give a one paragraph summary."

The claims in the quoted text are incorrect. According to multiple authoritative sources, the CFP National Championship game on January 19, 2026, was played between the Indiana Hoosiers and the Miami Hurricanes, not Duke Blue Devils and Indiana. Indiana defeated Miami 27-21 to win their first-ever national championship, completing an undefeated 16-0 season. The game was held at Hard Rock Stadium in Miami Gardens, Florida, and featured Indiana as the No. 1 seed and Miami as the No. 10 seed. Duke Blue Devils won the ACC Championship in 2025 but did not qualify for the College Football Playoff, finishing their season 9-5 after defeating Arizona State 42-39 in the Sun Bowl on December 31, 2025.
https://en.wikipedia.org/wiki/2026_College_Football_Playoff_National_Championship is given as one of the references as a footnote after the paragraph.

As I'm saying till I'm hoarse, relying for LLM training data for your knowledge is a high-risk proposition, as there is literally no indication if the output is non-sense (aka the so called hallucination) or really a statistical "recall" from training data.

Anybody with basic knowledge (they don't need to study the subject) can tell you.

Fewer (because good tools like these are not as common) can tell you that LLM works quite well for NLP tasks, and surprising well (if not always perfect) for orchestrating tasks.

Grey Goosed?

I think we're actually not far from the point where a good model with a body of work could write stories you couldn't distinguish from ones I wrote. That both impresses and depresses me.

Was this post written by you, or an AI trained on your posts? Forget that, how would I be able to tell if the response was generated by an AI purporting to be you?

edited to add: likewise my post. (Except who would bother?) Perhapse the presence of typos is now an indicator of genuineness.

Me, despite the comments about starting each paragraph with 'I'. Didnd't even use AI to spellcheck or proofread it. And no sentence in this response begins with that accursed letter! 🤪

And that's precisely what a competent ML simulator would say!