AI detectors are tools that are used to detect whether a piece of text is written by AI or not. AI detectors are based on language models similar to those used in the AI writing tools they are returning to detect. The model looks for two things in a text: perplexity & burstiness, the lower these variables are, the more likely the text is to be Al-generated.
Perplexity : a measure of how unpredictable a text is.
It refers to how well the model can predict the next word in a sequence of words. As you’ll probably know by now, AI-generated text is procedurally generated; i.e. word-by-word. AI selects the next probable word in a sentence from a K-number of weighted options in the sample.
Perplexity is based on the concept of entropy, which is the amount of chaos or randomness in a system.
• Al models generate text that reads smoothly and is more predictable.
• Whereas more creative language choices are made by a human
writer.
Burstiness : measure of variation in sentence structure and length
Burstiness measures how predictable a piece of content is by the homogeneity of the length and structure of sentences throughout the text. In some ways, similar to perplexity but on the level of sentences rather than words.
• Text with less variation has lower burstiness & text with high variation has high burstiness
Al tends to produce sentences of average length (say 10 - 20 words) with conventional structures. The reason why AI writing can seem monotonous whereas real people tend to write in bursts — we naturally switch things up and write long sentences or short ones; we might get interested in a topic and run on, propelled by our verbal momentum.
Are these detectors reliable?
In January, Open Al unveiled its own AI detector but acknowledged that it is 'not fully reliable.
According to the company's evaluations, the indicator correctly identifies 26% of Al written text as "likely AI - written while 9% of the human written text is incorrectly labeled as Al written.
A new scientist reports that a team used Al-based tools to rewrite AI-generated text with and without watermarks and fed it to several AI text detectors.
They found that the accuracy of the majority of the detectors was reduced to around 50%, leading to a significant drop in performance.
So we can say that the AI detectors are not reliable as the text generated by AI can be edited by the user and when the final text is checked by detectors results are not reliable
This post was written by Sakshi Sehrawat