Raschka AI Research builds DistilBERT AI text detector

Raschka AI Research Lab has released a comprehensive guide on building an AI text detector from scratch, utilizing a fine-tuned DistilBERT classifier. Inspired by Substack's recent integration of an AI detector in its user interface, the educational project demonstrates how small language models can be used to construct, evaluate, and deploy local verification systems. The resulting tool provides a local browser interface and an API that can be utilized by both human writers and autonomous agents.

The detector functions as a supervised classifier that outputs a score from 0 to 100, representing the estimated probability that a given text is AI-generated based on its training distribution. Beyond simply flagging machine-written content, the system can highlight individual text chunks to pinpoint specific areas that trigger the classifier. The methodology mirrors the Pangram models that power Substack's native detection features, building upon research into supervised classifiers, perplexity measures, and watermarking that dates back to 2023.

For AI practitioners, this project serves as a practical case study for building verifiers to guide small language models. By using the detector as a verifier, developers can train language models using Reinforcement Learning with Verifiable Rewards to generate text that actively avoids detection. This setup addresses the ongoing cat-and-mouse game of AI detection, where classifiers must constantly adapt to new generation patterns to avoid false positives.

Outside of training loops, the detector offers immediate utility for content moderation and personal writing. Writers can use the tool to polish their drafts with large language models while ensuring the final output does not sound overly processed. By setting a target score, such as zero percent AI-generated, users can leverage grammar checkers to improve readability without risking their content being flagged as spam.