2026-09-12 · post #6

A $0.38 AI-text detector

Harsh Gupta

It looks like a product on ChatGPT answers. Change the generator and it falls apart.

We trained a small classifier on Compute that reads an answer and says human or AI. On 4,502 held-out HC3 answers it scores 99.4% and flags 1.0% of the human text. On a 600-row RAID slice — GPT-4, Llama-chat, and human, none of it ChatGPT — accuracy is 48%. The headline GPU run cost $0.30. The whole issue was $0.38.

HC3 accuracy: 99.4%
Human false positives: 1.0%
RAID accuracy: 48.3%
Issue spend: $0.38

Stacked bars: DistilBERT flags 99.9 percent of ChatGPT answers on HC3 and 22.8 percent of GPT-4 and Llama answers on RAID, with 77.2 percent walking through — The number readers should remember. Same weights. Same 0.5 threshold. A different generator.

Human or ChatGPT, on purpose

This is not a Pangram alternative. Pangram is a product that has to survive new models, new prompts, and people who will try to fool it. This is a two-evening classifier on a 2023 ChatGPT corpus, plus one honest out-of-distribution check.

The training set is Hello-SimpleAI/HC3 English: 23,763 usable questions after dropping short answers, each with one human answer and one ChatGPT answer. The job downloads that on the machine. You do not upload it.

We split by question, not by answer, so a ChatGPT reply never sits in training while its human twin sits in test. Medicine is held out entirely — same generator, new domain. 20,268 questions stay for training, 2,251 for the in-distribution holdout, 1,244 for medicine.

Stacked bar of 23,763 HC3 questions: 20,268 train, 2,251 holdout, 1,244 medicine — Each question contributes one human answer and one ChatGPT answer. Medicine never appears in training.

A laptop baseline, then DistilBERT

Two classifiers, same labels, same splits:

TF-IDF (1–2 grams, 50k features) plus logistic regression.
DistilBERT (distilbert-base-uncased, 66,955,010 parameters) with a two-class head.

Loss is class-weighted cross-entropy. Accuracy is reported. The number that matters is the false-positive rate on human text: how often a person gets called a machine.

Flow diagram: answer text enters DistilBERT, which calls the text human or AI — On HC3 it almost always flags ChatGPT. On RAID most GPT-4 and Llama answers are called human.

Two-epoch chart with dots: train loss falling from 0.044 to 0.009, eval loss from 0.023 to 0.018, held-out accuracy rising from 99.22 percent to 99.42 percent — Two epochs. Eval loss kept falling. We did not pick a checkpoint from these curves — the reported numbers are the final epoch.

Easy on ChatGPT, then a coin flip

Headline run run_ea7b0d5c56f928808c3096a232e3e0bd on a RunPod A100-SXM 80GB. DistilBERT beats TF-IDF on the HC3 holdout and on medicine. Both collapse on RAID.

Threshold 0.5. RAID is 200 human, 200 GPT-4, and 200 Llama-chat documents with no adversarial attacks.
Split	Model	Accuracy	Human FPR	AI recall
HC3 holdout	TF-IDF	97.1%	2.4%	96.6%
HC3 holdout	DistilBERT	99.4%	1.0%	99.9%
Medicine	TF-IDF	97.0%	5.9%	99.9%
Medicine	DistilBERT	98.1%	3.5%	99.7%
RAID OOD	TF-IDF	49.3%	8.0%	28.0%
RAID OOD	DistilBERT	48.3%	0.5%	22.8%

Horizontal bars: DistilBERT accuracy 99.4 percent on HC3, 98.1 percent on medicine, 48.3 percent on RAID, with TF-IDF beside each — Accuracy is the wrong comfort. Medicine still looks solved because it is still ChatGPT. RAID is a coin flip.

Horizontal bars of human false-positive rate: DistilBERT 1.0 percent on HC3, 3.5 percent on medicine, 0.5 percent on RAID — The classroom number. DistilBERT flags 23 of 2,251 held-out humans on HC3, and one of 200 on RAID — while missing 77% of the new-generator text.

Side-by-side confusion matrices: HC3 is almost a perfect diagonal; RAID calls 309 of 400 GPT-4 and Llama answers human — The collapse, as counts. Steel is a correct call. Orange is a miss. RAID’s orange square is 309 GPT-4 and Llama answers that walked through.

Medicine still works because it is still ChatGPT. RAID does not, because GPT-4 and Llama are not the 2023 ChatGPT that wrote HC3. DistilBERT’s RAID accuracy is slightly worse than TF-IDF. It is also much less willing to accuse a human. That is the trade a detector actually has to make, and it loses on this slice.

What it still gets wrong on HC3

These are the examples the run printed from the in-distribution holdout: three humans it called AI, and three ChatGPT answers it let through.

Human, called AIreddit_eli5 · P=0.998

Radiation is defined as waves or particles emitted from an atom as it moves from a higher energy state to a lower energy state. This comes in several forms: heat, light, beta…

Human, called AIopen_qa · P=0.998

Flo is a fictional character who appears in commercials for Progressive Insurance. She is played by actress and comedian Stephanie Courtney…

Human, called AIreddit_eli5 · P=0.994

Bitcoin is much like a digital form of gold. In order to mine gold, you need to spend time and money to collect the ore and refine it…

ChatGPT, called humanfinance · P=0.395

If the underlying stays below the strike price, the put option will not be exercised and will expire worthless. This means that the value of the put option will be zero.

ChatGPT, called humanreddit_eli5 · P=0.202

No, the Earth is not getting bigger. The Earth is made up of solid rock, and while the surface of the Earth can change over time…

ChatGPT, called humanfinance · P=0.114

Why can't poor countries just print more money

The false positives are tidy encyclopedia sentences. The misses are short, or they look like a question that leaked into the answer field. HC3 is not a clean product dataset. The 99.4% is agreement with that dataset, including its mess.

Train it on Compute

One file. Hand compute.cx/SKILL.md to your agent, plus the specials in this post’s overlay.

curl -fsSL https://raw.githubusercontent.com/theoriclabs/letsusecompute/main/posts/ai-text-detector/train.py -o train.py
curl -fsSL https://compute.cx/install.sh | sh
compute setup
compute credits add 10

compute run train.py::train --gpu cheap --dry-run
compute run train.py::train --provider runpod --gpu A100-PCIe-80GB --timeout 2400 --wait

Dry-run only prints the upload plan. The real command quotes a GPU before spend. --gpu cheap is the issue default. On this account it quoted the decorator SKU and Vast had no interruptible offers, so the sample ran on RunPod A100-PCIe and the headline on A100-SXM. A short sample is enough to check the metric code first:

compute run train.py::train --provider runpod --gpu A100-PCIe-80GB --timeout 2400 --wait --args '{"max_train": 2000, "max_eval": 400, "epochs": 1, "ood_per_model": 40, "ood_scan": 20000}'

Want to read the script? train.py.

The weights landed this time

Sample run run_3eaf70692b424b122f8656610c28d501 cost $0.08. Headline run run_ea7b0d5c56f928808c3096a232e3e0bd billed 9 minutes of training and 1 minute of boot, $0.30. Both returned ok: true and both persisted a five-file artifact — detector.pt is the classification head, not the full 250MB DistilBERT folder. There is no Hub checkpoint. No hf secret was set.

Reports: rpt_1f125aff10b2f5922cf0cb077d8b4eeb (cheap quoted Vast RTX-3090, no interruptible offers), rpt_49171d9a8d52914e5af23341cb798aea (same after switching the decorator to RTX-6000-ADA, which the catalog marked available), rpt_b1c5888080691f47853bbab6fd1b4d2b and rpt_4daac4e75ba231e1857f282495c32549 (two RunPod A100-PCIe creates returned HTTP 500 after the sample had succeeded on that SKU).

If artifacts land on your run:

compute artifacts list <run_id>
compute artifacts get <run_id> <artifact_id> <version> --out ./weights

Storing a Hugging Face token does not put it on the machine. Secret.from_name("hf") on train_and_push does.

compute secrets set hf
compute run train.py::train_and_push --provider runpod --gpu A100-PCIe-80GB --timeout 2400 --wait

What this does not claim

It does not detect “AI text” in general. It detects 2023 ChatGPT answers that look like HC3, and it does that very well. It does not locate a generator, attribute a document to a person, or survive paraphrases, decoding tricks, or models released after the training data. RAID here is 600 streamed rows, not the full benchmark. Medicine is still ChatGPT. The false-positive rate is the number that would matter in a classroom or a newsroom, and 1% of 2,251 held-out humans is still 23 people.

That is as far as $0.38 got us from Pangram.