Doug Turnbull Uses LLM Hallucinations to Tag Content

Categorizing large archives of content using large language models often presents a scaling challenge when dealing with massive taxonomies. Developer Simon Willison highlighted a creative solution designed by search expert Doug Turnbull to address this issue. Instead of forcing an LLM to select from a massive list of pre-existing categories, Turnbull suggests letting the model freely generate, or 'hallucinate,' the ideal tags for a piece of text, and then mapping those outputs back to the existing vocabulary using vector embeddings.

Traditional classification workflows require feeding an entire taxonomy into an LLM prompt. For extensive systems, such as Willison's blog which contains 1,856 unique tags, this approach quickly exhausts context windows and increases API costs. Turnbull's method bypasses this limitation by prompting the model to invent novel classifications from scratch. To ensure the model outputs tags in the correct structural format, practitioners can provide a few-shot prompt showing examples of the desired taxonomy shape, such as multi-tiered product categories like furniture, decor, or kitchen organization.

Once the LLM generates these hypothetical tags, the workflow uses vector embeddings to calculate the semantic similarity between the generated terms and the actual taxonomy corpus. The system then selects the closest existing tags to apply to the content. This allows practitioners to maintain a consistent, pre-defined vocabulary while leveraging the creative reasoning of generative models.

For AI practitioners and database administrators, this hybrid approach drastically reduces prompt complexity and token consumption. It eliminates the need to constantly update prompts with evolving taxonomies, making it highly scalable for enterprise e-commerce catalogs, digital libraries, and content management systems. By decoupling the generation phase from the strict classification phase, developers can achieve highly accurate tagging with minimal computational overhead.