Home Β» Forum Β» Author Hangout

Forum: Author Hangout

AI detection in higher education

bison9 🚫

Just so you know, the snake oil vendors are generally not the ones that carry the pain of their products "quality":

"We will continue our policy of not allowing the use of any AI detector as the sole source of support for a
case against a student for academic misconduct in the future."

Universities find it unfunny of losing cases. In this case because of Turnitin's AI detection a marketleader in the academic field. (Notice that they have by all measures the same snakeoil that GPTzero has, but because they also have the moat of all the academic papers being handed over to them, they can finetune the corpus better) Yet they officially admit a false positive rate of 4%.

Which leads to the WSU losing 33% of their AI related "violation of academic integrity" hearings, thus canceling the contract.

https://provost.wsu.edu/documents/2026/02/cancellation-of-turnitin-ai-detection-software_memo-to-instructors_provost-office_spring-2026.pdf/

Nope, AI detection is snake oil.

Bison

bison9 🚫

@bison9

"Instructors are in a difficult position due to the technological revolution that AI has introduced…AI can earn passing grades on many types of assessments used at universities, which calls into question the validity of many traditional methods of assessing student learning," he said. "This means instructors need to do the hard work of reengineering assessments so that they include elements that are secure and AI-resilient."

Trivial, Dr. Watson, said Sherlock.

Go back to a Humboldt system of higher education. Oral exams, written exams open book by default, and a completely different level of TA/assistent profs to student ratio.

As studying in my youth was pre-Bologna in the EU.

Before we were infected by US neoliberal ideas in Higher Education, sigh.

Replies:   Grey Wolf
Grey Wolf 🚫

@bison9

Oral exams are very difficult to impossible for large classes, and most universities could not function without large classes. How are you going to give an oral exam to 120 people (I had ~10 classes at that size or so) or 600 (I know people who had ~10 600+ classes, at a different university)? Maybe if you had 60 TAs, but then you have to figure out how to standardize grading between them.

On the other hand, written exams taken in class solves much of the problem, barring outright cheating (e.g. radios, use of AI on smart devices, etc). It's also tricky to police cheating in a 120-person class, but that's been done for years.

That doesn't get you serious papers, though. You can't develop and write a research paper in class (unless it's all you do in class, and perhaps not even then).

As of now, the ability to write a research paper is still important. Perhaps, a decade from now, we'll look back and think, 'How quaint! We used to think it was important to be able to do trigonometry with a slide rule, too!' But that's then, not now, and I'm not sure we'll get to that point. It's probably closer to calculators. It's important to know how to get to a result without a calculator, but allowing calculators makes far more sense for most exams, because that's what graduates will do, day-in and day-out.

An awful lot of this works out to whining that things are hard - not on the part of the students, but on the part of instructors and administrators. I've seen far too many cases of students who are either failed or expelled for reasons of 'academic integrity' because an 'AI detector' showed the use of AI, notwithstanding that the student kept a detailed audit trail of their development of the work.

Perhaps it should go both ways? Use AI inappropriately and get caught cheating - suffer serious consequences. Use an AI detector, blindly trust its results, and be proven wrong - suffer serious consequences. Right now, part of the problem is that it's relatively consequence-free for 'the system' to falsely accuse a student of cheating. Even if the student successfully sues to overturn the cheating accusation, there's little repercussion for those who made the accusation, nor the administrators who forced the use of the 'AI Detector.'

Replies:   awnlee jawking
awnlee jawking 🚫

@Grey Wolf

Oral exams are very difficult to impossible for large classes, and most universities could not function without large classes. How are you going to give an oral exam to 120 people

My secondary school managed that just fine with oral foreign language exams. If universities can't manage it, what do they spend all the tuition fees on?

A university qualification is supposed to provide some sort of confidence that the possessor knows the material well enough to apply it in the real world. If they can't stop students cheating by using AI, their degree awards are effectively worthless.

AJ

Replies:   Marc Nobbs  Grey Wolf  bison9
Marc Nobbs 🚫

@awnlee jawking

My secondary school managed that just fine with oral foreign language exams.

My wife is a French Teacher. Typically has 1 or 2 GCSE classes a year, so between 30 - 60 students. Each exam takes about 10-15 mins. That's between 5 and 15 hours of sitting in a room with a single student conducting their exam. If that's all she does during a school day, with someone else covering her other classes, that's up to 3 teaching days.

And that doesn't include marking. The recordings are sent away to be marked. She's also done this - marking other schools' scripts for AQA. Each script can take twice as long as the recording to mark, as she frequently rewinds and replays each one.

That's GCSE level. A-level orals are closer to 20 - 25 mins, although there are considerably fewer candidates at A-Level. I'd assume university-level orals would need to be even longer, but there would usually be many, many more candidates.

Orals are also only one of four components of the Languages exams - they still need to sit listening, reading and writing exams, so even that's not really a fair comparison.

Replies:   awnlee jawking  bison9
awnlee jawking 🚫

@Marc Nobbs

I guess orals have been improved out of existence. My secondary school insisted every pupil learn a foreign language, and some opted for multiple languages. The vast majority would progress to qualification level.

Oh well, orals by chatbot. Sigh!

AJ

bison9 🚫

@Marc Nobbs

Actually, our oral exams generally were done publicly.

With the next candidates waiting, thus having the option to redirect questions, eg asking to see if somebody can answer a more advanced question. I remember me answering such, and the examiner joking back "you guessed", and I explained to him yes and no, I deducted it like this. No, I hadn't known it by heart. Got me a B equivalent on a rather brutal and generally hated exam.

And public, as others as there was room, generally students (as it was perfect exam prep) listening to the exam, and acting as witnesses.

Technically the examiners (back then all of them civil servants) got even a small per head exam fee.

Grey Wolf 🚫

@awnlee jawking

(The first part of this assumes Mark Nobbs' comments) In terms of time, that's what I was thinking. For a secondary school foreign-language class (~24 people or so), that's perhaps three hours. That's still very time-consuming, if you're going to have multiple exams, but manageable.

Scale that to ~200 people, and when is the professor expected to actually teach? You can throw TAs at the problem, but then you have a number of different graders with potentially different standards.

And that's language fluency, which is relatively amenable to orals. History can be tested that way, but I've had exams that took three hours of nonstop writing. Maybe that's an hour of speaking. For even a 16-person class, when will that happen?

Orals can be great. I've passed Computer Science orals to be admitted as a doctoral candidate (a degree I did not actually complete, mostly because my experience suggested I didn't want the jobs PhD's actually get). That was an hour-long process with months of studying. Entirely worth it - even not getting the degree, I'm glad I did it - but not something you can expand to the average class.

How does one do orals for calculus or differential equations? Thermodynamics?

Stepping back, I'm not entirely convinced that we're not asking the wrong question. Are we essentially saying the equivalent of, 'How can we verify that people can write and spell if we all spellcheckers? We need tools to detect their use!' Or, 'How can we verify that people can properly use slide rules if they insist on using calculators?' Replace examples as necessary.

A university qualification is supposed to provide some sort of confidence that the possessor knows the material well enough to apply it in the real world.

If the student will be using AI in 'the real world,' the university would not be providing any confidence that the possessor knows their use well enough to apply it. Increasingly, the use of AI is required in a number of fields. One may dislike that, and there are serious societal problems we'll need to address along the way, but trying to graduate people in Computer Science without exposure to using AIs for coding would likely be malfeasance at this point. Most employers will expect a junior programmer to lean on AIs.

That doesn't mean we throw up our hands. I would still want a CS grad to be able to explain the Halting Problem without reference to notes or having an AI write the paper. But that's something you can test in a class. I would want them to have a lot of concepts one can test for in written form without use of an AI.

The problem circles back around to papers, though. Relatively few people's 'real world' jobs involve writing papers. People should know how to communicate effectively, but bad communication is bad communication. Perhaps the focus is on whether the output is good, not the tool.

If a current-generation AI can do the job at the level of a university graduate in that field, it will. No one's hiring people who can be entirely replaced by an AI. Universities should be teaching things AIs can't do, by and large.

Yes, that may mean wrenching overhauls in what we teach, how we teach it, and how we grade it. That sucks, in a lot of ways. But sticking our collective heads in the sand and replying on AI detectors to keep reality from intruding may be a fool's errand.

garymrssn 🚫

@Grey Wolf

No one's hiring people who can be entirely replaced by an AI. Universities should be teaching things AIs can't do, by and large.

With the full knowledge of being repetitious, AI is a tool,a labor saving device. Every labor saving device invented has done away with somebodies job. I predict that AI will be the most devastating white collar job ending device of the foreseeable future. In the past we have created new jobs that filled the gaps in the market for those displaced by new technology. However once corporations and stock holders realize the savings potential of AI they will adopt it faster than alternate jobs for unemployed white collars can be created.
Maybe some of you can see a light. I only see an oncoming train.
I would welcome any knowledgeable rebuttal.

Gary

Replies:   Grey Wolf  bison9
Grey Wolf 🚫

@garymrssn

This is not really a knowledgeable rebuttal. I suspect - but absolutely do not know - that the result will be job growth in areas we simply can't predict. But it might be something else entirely (the rise of Universal Basic Income, for instance).

From what I'm seeing right now, companies are quickly finding that the majority of jobs they thought existing AIs could do, they can't. Maybe that will change over time.

It's also entirely possible that an AI that can do job X is ridiculously more expensive than person Y who can do job X across a wide range of jobs (we're starting to see that, too). However, an AI that can assist person Y in doing job X better is cheap. That may be how things sort out.

Replies:   Marc Nobbs  garymrssn
Marc Nobbs 🚫

@Grey Wolf

It's also entirely possible that an AI that can do job X is ridiculously more expensive than person Y who can do job X across a wide range of jobs (we're starting to see that, too). However, an AI that can assist person Y in doing job X better is cheap. That may be how things sort out.

I think it is exactly what will happen. Take Law, for example. I work at a law firm (although I'm a bean counter, not a lawyer), and we've already discussed this. AI will assist our lawyer, not replace them, because without the human checking and confirming what the AI has done, the Professional Indemnity premium would almost certainly be too expensive. AIs make mistakes, it says at the bottom of every thread in the ChatGPT UI. Lawyers who make mistakes are very expensive for all involved. Therefore, AI Lawyers carry far too much risk to be relied on completely.

Replies:   bison9
bison9 🚫

@Marc Nobbs

Actually yes, so do humans.

Citation errors are actually e.g. in academia common, and depending upon field and study and methodology, numbers cited go up to 33% error rates in published papers they checked.

That's because they lump together wrong pagenumbers, logic fallacies, wrong citations, invented citations, plagiarism.

Only the last two generally cause troubles.

Similar in legal briefs β†’ a certain level of wrong is generally tolerated, just don't start invent the citations wholesale.

Now the urrent "state of the art" uses AI in a way that hallucinates citations. And e.g. the legal tools for citation checking come exactly from the database vendors who sell the legal case law databases, and they are a bit on the crude, suboptimal side, but hey, it's their data monopoly, and by US doctrine on monopolies that means their citation checkers must be the best too.

As it happens there is no reason for this, I'm just writing a grant request on that topic this weekend, so I a bit irritated with the world currently.

AI system

garymrssn 🚫

@Grey Wolf

knowledgeable

I meant reasonable but it took me four tries to come up with the word I used and I forgot that I could use a thesaurus to look up a word I forgot.
Thank you for the reply.

bison9 🚫

@garymrssn

Cough.

AI/LLM are all literally running at an huge deficit currently.

No real way out of it, and being dependant on half a dozen suppliers for your "workforce" sucks, supply chain-wise compared to even irritating, union organized humans.

The what I call the "American way", collect all data, throw huge compute on it, hope something good comes out of it tends not to deliver, 2026 is the year of the 'AI value consultants'. Linking the value to the spend on tokens backfired like July 4th firework.

awnlee jawking 🚫

@Grey Wolf

Relatively few people's 'real world' jobs involve writing papers.

The further up an organisation's hierarchy that someone progresses, the greater the importance of being able to write papers - white papers, green papers, and any number of other colours.

If they're ghost-written by hallucination-prone AIs, when company executives use them to decide how and where to allocate company resources, the consequences could be expensive.

AJ

Replies:   Grey Wolf
Grey Wolf 🚫

@awnlee jawking

I agree and disagree. There are people who write those, definitely - been there, done that. It's seldom one person, though, and they should be thoroughly fact-checked, not via AI.

But this goes back to what I said. If the AI can write a accurate, factual paper that meets the organization's needs more cheaply than a human can, it will - and the skill of writing that paper will be functionally obsolete. But, if that AI-written paper has expensive consequences, the job of the human will become 'avoid expensive consequences.' Not eliminated, just updated.

Replies:   garymrssn
garymrssn 🚫

@Grey Wolf

This article from Gartner may be an indication of what cooler heads are thinking on this subject. At least it is an interesting diversion from the AI hype I've been seeing and it does tie in to the higher education topic under discussion. It expresses the opinion that some of those replaced by AI will have to be rehired over the next few years.

https://www.gartner.com/en/newsroom/press-releases/2026-09-09-gartner-identifies-four-shifts-shaping-the-future-of-work

Replies:   awnlee jawking
awnlee jawking 🚫

@garymrssn

Pffft, some companies are already rehiring their AI casualties

https://www.bbc.co.uk/news/articles/cgrkd41n2v9o

AJ

bison9 🚫

@Grey Wolf

Trivial. The way we still do oral Matura (high school exit and university maturity exam) in Austria.

You give the candidate an exercise (or 2/3 and let her/him/them chose) and let her sit in the back working on the solution. β†’ 15-30m when you are done with the candidates before you call the victim to the board and watch the solution being presented.

That's basically how every "class" in a tech (STEM) subject works in Austrian unis and has worked for 40 years, video calls didn't change much:

- you get an exercise sheet with 5-10 examples
- depending on the ECTS it's weekly or bi-weekly. You tick the examples you managed. In the online, moodle be thanked you often have to upload your solutions before class.
- than the TA calls (randomly) victims to present.

Variations exist on when you are allowed to tick, e.g. even if your solution is not perfect, or how random the selection is, but it worked like this 1992 and does in 2026, on half a dozen Austrian unis.

prep β†’ presentation.

The uni version often also involved discussions, we are at least at the end of the BSc/in the MSc expected to do easier deriviations/integrals in the head.

There is a reason why our examiners shrug about LLM.

bison9 🚫

@awnlee jawking

Well, Austrian universities managed it fine.

E.g. in Medicine. So because formally it was a M.D./Ph.D. study course from semester one, each of the subject matter closing exams were an oral partial rigorosum exam. Offered regularly by all the Profs/Ass-profs with 10 or so slots once a month or twice.
BTW, public by law, and highly recommended to listen how your examiner examines, as part of your exam prep.

For written exams (we had those too) student unions literally collected old exams so you knew what to expect.

But yes our unis are even today much less profit orinted than US oriented ones.
(actually, they are still basically free with only the student union fee being mandatory. OTOH they suck when it comes to accommodating part-time studying, sigh)

Replies:   awnlee jawking
awnlee jawking 🚫

@bison9

E.g. in Medicine. So because formally it was a M.D./Ph.D. study course from semester one, each of the subject matter closing exams were an oral partial rigorosum exam.

I thought all doctors had to pass oral examinations. But in 'The Good Doctor', Shaun is allegedly a fully qualified doctor who 'doesn't answer questions.' I have to wonder how realistic that was. Can American doctors qualify without orals?

AJ

Replies:   Michael Loucks
Michael Loucks 🚫
Updated:

@awnlee jawking

Can American doctors qualify without orals?

Yes. The MLE does not have an oral component. Many Board Certifications do require oral exams, and some states impose additional requirements, but the MLE is a written (i.e on computer) test.

John Demille 🚫

@bison9

Yet they officially admit a false positive rate of 4%.

So basically they correctly identify 96% of all cheating. I would say that is a remarkable success rate.

Good things places like SOL and ZBookStore have in their terms of service that they can reject a work for any reason. They can't be sued for a false positive.

Also, it's not exactly strange that current academia doesn't fight for rigorous integrity. They have long been "anything goes as long as you pay use a huge tuition."

Witness this glorious subject:

Glaciers, Gender, and Science - A Feminist Glaciology Framework For Global Environmental Change Research.

https://www.scribd.com/document/659281633/Glaciers-Gender-And-Science-A-Feminist-Glaciology-Framework-for-Global-Environmental-Change-Research

Dominions Son 🚫

@John Demille

@bison9

Yet they officially admit a false positive rate of 4%.

So basically they correctly identify 96% of all cheating. I would say that is a remarkable success rate.

Faulty reasoning on two fronts.

1. It is exceedingly unlikely that the false negative rate is 0%, especially with such a low false positive rate, so even if the false positive rate is only 4%, the correctly identified cheating would still be below 96%

2. "they officially admit a false positive rate of 4%." So, they admit to a false positive rate of 4%. The real false positive rate is almost certainly higher.

julka 🚫

@John Demille

So basically they correctly identify 96% of all cheating.

I would love to know how you're arriving at this conclusion, because "Yet they officially admit a false positive rate of 4%." does not support it even a little bit.

Marc Nobbs 🚫
Updated:

@John Demille

So basically they correctly identify 96% of all cheating. I would say that is a remarkable success rate.

No. This means 96% of what they flag as cheating is actually cheating, not that 96% of all cheating is caught. We cannot know the 'false negative' rate from this stat.

However, 4% false positives seem low until you're part of the 4%. That's 40 out of every 1,000 candidates. 400 out of every 10,000. That's huge. Particularly for something that's potentially career-ending.

Apply the same rate to, say, the death penalty. Is killing 40 innocent people out of every 1,000 acceptable? No. It's not.

Dinsdale 🚫

@bison9

A reputable German computer magazine did some testing of "Pangram 4" with various texts, some old and some new.
Running the program multiple times on the same text would get wildly different results.
At that point they copied-and-pasted the text into 4 different document types, RTF, DOCX, PDF, text. They then ran each test 5 times. The values for PDF and DOCX showed enormous variation across the tests, the values for RTF and text were more internally consistent, although just as frequently totally wrong.

Deliberate AI Slop would be declared to be human with a "high degree of confidence", and virtually everything was with a "high degree of confidence".

Iam4u1ly 🚫

@bison9

I heard that Turnitin can't identify AI content effectively. It is used only to detect plagiarism, according to many faculty members at a Top-10 UK university. And the rules have not been tweaked yet to impact the grades of those who use AI.

fohjoffs 🚫
Updated:

@bison9

I have three STEM degrees, the last of which was granted in 1991. There were 'expert systems' during that period, but LLM-type AI was only theoretical, although much discussed. The concept of LLMs, during the 80s through the early 2010s was simply waiting for advent of mass-produced, affordable processing power. None of this shit is new, nor is it novel.

But, back then, this shit was the solely the realm of the engineering community. The unwashed masses, and the general media, had no conceptional framework to grok what was coming, and any talk of such was regarded as pulp sci-fi by the stupid, which comprises the majority of humanity (yes, that is elitist. and it is truth).

So here we are, with the unwashed ignorantia having easy access to a technology well beyond their comprehension. It will get much worse before it gets better.

Personally, I love the unholy fuck out of this LLM shit. It has given my cranky geezer mind a second lease on my professional life. And these same LLMs may well destroy many other's lives. Rather Darwinian, don't ya think?

Replies:   awnlee jawking
awnlee jawking 🚫

@fohjoffs

Rather Darwinian, don't ya think?

Darwinian is not being able to check a restaurant bill, despite having a maths degree, because you don't have a calculator.

AJ

Replies:   fohjoffs
fohjoffs 🚫

@awnlee jawking

I do not understand the comment. Non sequitur? Never heard or experienced such a thing. N=1? But back to the subject.

I am not a biologist. I am but a lowly engineer, beating electronic components and code into a cruel submission to meet my needs and whims, or the unruly specs of a customer. As such, my view is aided, and clouded, by the mathematics and physics within my intellectual tool box. Thus, I believe the recent advent of commonly-available high-performance hardware, that is able to quickly parse huge data sets and efficiently execute LLMs, will divide and isolate humanity in many ways. It will also join a few elements of humanity, but in fewer and limited ways.

We introduced machinery that effects 'the' ecosystem. Our machinery has changed humans at the societal and cultural levels, and now affects the structure of the human organism. These changes are no longer limited to geologic time scales.

Technology forms our enviroment. The ability to adapt to a changing enviroment is a central premise; ie, an evolutionary force. Genetic variation seems to depend on enviromental stressors, as a precursor to evolution. I have no idea how stressors induced by LLM stuff will play out over the next few hundred years. I've yet to note any biology wonks that have taken on any reasonable study of this subject.

Humans appear to be unique in that we can both contain and create genetic variation, at least in the lab (CRISPR) for some limited subsets of life. But aside from some scientist hacking away at mitchondiral DNA, humans do create and change the enviroment through our computational prowess. So the question I have for anyone foolish enough to read this crap, 'Is the thing popularly referenced as 'AI' a sufficient stressor to effect species-level survivability via natural selection of individuals?'. I dunno, but my semi-logical conclusion is that 'AI' induced selection is inevitable. Darwin will win again; but not in ways he had imagined.

FWIW, at 70, I can do arithmetic calculations in my evil little mind, to include irrational numbers. And yes, one of my degrees is in Mathematics (not 'Maths', whatever old-world dogshit that may be).

jimq2 🚫

@bison9

During the 80's my brother was at UC Berkeley and the professor noticed most of a page that a direct quote from her own thesis from 8 years earlier. The student didn't notice the name on the thesis was the professor's maiden name.

Replies:   bison9
bison9 🚫

@jimq2

oops.

That's what you have deep research assistants today for, that track a person's CV in detail, notice odd surname changes of persons in their historic artifacts, and also remark on odd fellows who are oddly publicity shy for their assumed role.

Back to Top

 

WARNING! ADULT CONTENT...

Storiesonline is for adult entertainment only. By accessing this site you declare that you are of legal age and that you agree with our Terms of Service and Privacy Policy.


Log In