Introduction

Large language models have revolutionized various tasks, from drafting emails and summarizing reports to writing code and tutoring students. Yet, they sometimes generate content that is incorrect or unfounded. This phenomenon, known as hallucination in AI, occurs when a model invents a citation, misstates a fact, or claims certainty about something it cannot actually know.

This article aims to clarify why language models hallucinate, identify when this occurs most frequently, and offer strategies to mitigate it. We will build upon the key insights from OpenAI’s post, “Why language models hallucinate,” while presenting them in a way that is accessible to readers, AI enthusiasts, and professionals alike. Where applicable, we will refer to independent research and practical methods that can help minimize hallucinations in real-world systems.

What do we mean by hallucinations in language models?

In common parlance, hallucination refers to instances where a model generates text that lacks grounding in verified facts or the information given. Researchers categorize hallucinations into two primary types:

• Intrinsic hallucinations: These occur when the model makes statements that contradict the source or context. For instance, summarizing a document incorrectly by introducing details that aren’t present.
• Extrinsic hallucinations: These involve confident assertions that cannot be corroborated or are unsupported by the user’s prompt or the model’s resources. A classic example includes a model fabricating a bibliographic reference that seems legitimate but does not actually exist.

While the terminology may vary across studies, the underlying principle remains the same: hallucinations yield fluent, confident outputs that are not reliably accurate or grounded. A comprehensive survey offers an insightful overview of these terms and various evaluation methodologies [Ji et al., 2023].

Why do language models hallucinate?

Language models do not possess beliefs, nor do they verify facts by default. They are designed to predict the next word in a sentence based on a vast corpus of text. This design leads to several predictable failure modes:

1) Next-token prediction prioritizes plausibility over proof

The model learns to generate likely continuations based on patterns that sound correct. In open-ended situations, there may be numerous plausible, yet inaccurate, responses that commonly appear online. Without proper grounding or retrieval, the model may select a fluent but erroneous answer, particularly in knowledge-intensive tasks or when summarizing beyond the source material.

2) Issues with data distribution and training gaps

Models are trained on data collected at a specific time and distribution. As such, queries about events occurring after the training cutoff or about niche topics that are underrepresented in the training data may yield unreliable generalizations. This situation, known as distribution shift, can manifest as guesswork or confident inaccuracies when up-to-date or domain-specific context is lacking.

3) Decoding settings can compromise accuracy for diversity

Sampling methods like temperature, top-k, and nucleus (top-p) sampling can increase the diversity of outputs, but they may also lead to increased error rates. A higher temperature often results in more creative and unpredictable text, thus heightening the potential for hallucination. Research indicates that decoding settings significantly influence output quality and factual accuracy [Holtzman et al., 2020].

4) Long contexts can amplify errors

In lengthy prompts or multi-step tasks, small mistakes can accumulate, leading to larger inaccuracies. If the model misinterprets information early on, subsequent steps may reinforce incorrect beliefs. When tasked with generating extensive responses in one go, the model carries forward numerous unverified assumptions, increasing the risk of deviating from the truth.

5) Following instructions isn’t synonymous with factual accuracy

Post-training techniques like supervised fine-tuning and reinforcement learning from human feedback (RLHF) help models align better with user intent and enhance assistance. However, merely following directions does not ensure correctness. In fact, signals that indicate helpfulness can reward fluent, confident answers over cautious ones. Although RLHF and similar methods have bolstered reliability, they should complement, not replace, mechanisms for grounding and verification [Ouyang et al., 2022], [Bai et al., 2022].

6) Inadequate calibration leads to overconfidence

Even when a model’s internal signals indicate uncertainty, it may still present definitive answers. Better calibration aligns output confidence with the model’s actual likelihood of being correct. Numerous studies indicate that when models express uncertainty, their hallucination rates diminish [Kadavath et al., 2022].

7) Ambiguous or adversarial prompts exacerbate hallucinations

Prompts that are vague, adversarial, or designed to evoke speculative responses can increase the incidence of hallucinations. Instructions that promote verbosity or creativity without constraints can also lead the model to make unwarranted assumptions.

When are hallucinations most likely to occur?

• Open-ended questions that do not have a definitive answer.
• Requests for niche, time-sensitive, or proprietary information that may not be adequately covered in the training data.
• Long-form generation, especially in tasks that require summarization or synthesis beyond the available source materials.
• Creative decoding involving higher temperatures or aggressive sampling techniques.
• Few-shot tasks where the provided examples may be noisy or biased.
• Complex reasoning tasks without access to external tools for math, coding, or data retrieval, which would enhance accuracy.

How to Measure Hallucinations

While there is no single perfect metric, several effective methods exist:

• Grounded QA and knowledge tasks: Assess factuality against trusted sources or authoritative answers. Datasets like TruthfulQA evaluate whether models replicate common inaccuracies or misinformation [Lin et al., 2022].
• Faithfulness in summarization: Analyze model-generated summaries against source documents, penalizing unsupported content through methods that range from human review to automated checks leveraging entailment or citation overlap.
• Factuality scoring and claims verification: Metrics like FactScore segment outputs into individual claims and validate each one against trusted references or retrieval systems [Min et al., 2023].
• System-level evaluations and red teaming: Utilize domain-specific tests, adversarial prompts, and case libraries to monitor hallucination rates over time. Open-source frameworks like OpenAI Evals facilitate customized task evaluation and monitoring [OpenAI Evals].

How to Reduce Hallucinations: Practical Strategies

While there is no one-size-fits-all solution, adopting a multi-faceted approach can greatly enhance reliability. Here are several strategies you can implement within your production systems.

1) Enhance Models with Retrieval Systems

Integrating retrieval-augmented generation (RAG) equips models with relevant excerpts from knowledge bases or the web, allowing them to cite and reason with verifiable sources. Instead of solely relying on internal memory, models can leverage retrieved evidence, which improves factuality in knowledge-rich contexts and reduces hallucinations [Lewis et al., 2020]. Utilizing dense passage retrieval combined with hybrid retrieval often enhances recall [Karpukhin et al., 2020].

• Pair RAG with clear directives like: “Only use the documents provided to answer. If the documents do not suffice, state ‘I don’t know.'”
• Favor concise, attributed quotes and a summary that authenticates specific excerpts.
• Regularly update your index to reflect timely topics.

2) Utilize External Tools for Precision Tasks

Provide models with access to calculators, coding environments, databases, or APIs, and instruct them to utilize these resources. This shifts the accuracy responsibility to the appropriate tool and diminishes the reliance on conjecture. Tool access is particularly beneficial for domain-specific tasks involving math, structured data, and validation of identifiers.

• Define functions with strict architectures and validate incoming arguments.
• For data analysis, route natural language queries to a SQL generator and execute the query instead of letting the model estimate figures.
• Maintain logs of tool usage and present results to users for transparency.

3) Optimize Decoding for Reliability

In scenarios where accuracy is critical, lowering temperature settings and narrowing sampling options can minimize random errors. Beam search may be advantageous in some cases; however, for factual accuracy, a conservative temperature with nucleus sampling usually proves effective. Consider implementing a two-pass process where the model drafts a response, followed by verification or citation addition.

4) Encourage Models to Reason and Self-Check

Structured prompts can significantly lower hallucination rates:

• Step-by-step reasoning: Request the model outline its thought process before answering to avoid leaps in logic and inaccuracies.
• Self-consistency: Generate multiple independent rationales and apply a majority vote; this often enhances the accuracy of reasoning tasks [Wang et al., 2022].
• Critique then revise: Challenge the model to evaluate its own output and rectify issues prior to delivering the final response.
• Explicit abstention: Include directives and examples where the model acknowledges uncertainty by stating “I don’t know” or requests additional information when needed, thus improving its calibration [Kadavath et al., 2022].

5) Use Schemas to Constrain Outputs

For structured results, implement JSON schemas, enumerated types, and strict parsing methods. Constraining valid outputs minimizes the chance of erroneous data entries or formats. Combine schemas with validation and post-processing to prevent or flag suspect outputs before they reach users.

6) Train Models for Honesty and Accuracy

Post-training procedures should reward precision and truthfulness alongside fluency. Supervised fine-tuning on grounded Q&A, training that prioritizes cited outputs, and RLHF that encourages refusal to speculate can enhance both reliability and honesty [Ouyang et al., 2022], [Bai et al., 2022]. These approaches should work in conjunction with retrieval and verification techniques.

7) Implement External Verification Mechanisms

One-time validations are insufficient. Introduce verifiers to scrutinize claims, citations, or code outputs. Utilize a secondary model or a different prompt to assess the initial model’s output. Cross-reference named entities, dates, figures, and quotations with retrieval systems. Automated claim extraction coupled with verification can catch a significant number of errors before they are presented to users [Min et al., 2023].

8) Design User Experiences to Promote Accuracy

Product design choices can dramatically affect hallucination risk:

• Surface sources: Display citations or specific segments utilized. Allow users to review the source material.
• Communicate uncertainty: Use confidence indicators or phrases like “Based on the sources provided…” where suitable.
• Facilitate easy corrections: Provide user feedback controls, allow quick retries with different settings, and incorporate mechanisms for reporting issues.
• Default to cautious behavior: In sensitive areas (medical, legal, financial), enforce citation requirements and block unsupported statements. Prioritize tool usage and retrieval, and incorporate disclaimers or human review where necessary.

Developing a Comprehensive Playbook

1. Clearly define hallucination for your specific use case (e.g., unsupported claims, incorrect citations, incorrect numbers, contradictions to source material).
2. Implement evaluation using a small but representative test dataset and monitor hallucination rates over time, including grappling with edge cases.
3. Utilize RAG for knowledge-intensive tasks and keep your database current. Require citations from retrieved sources.
4. Use external tools for math, structured queries, identifiers, and anything requiring high precision.
5. Refine your prompts: instruct the model to abstain when uncertain, ask for detailed reasoning, and require verification steps.
6. Implement output constraints using schemas and validate before results are delivered.
7. Optimize decoding for reliability in high-stakes situations; consider a two-tiered approach involving a verification pass.
8. Close the feedback loop: gather user input, retrain on quality counterexamples, and continuously monitor performance.

Key insight: Hallucinations are an expected side effect of next-token prediction. Their occurrence can be reduced by equipping models with proper tools and incentives for verification, citation, and abstention.

Examples and Patterns

Example 1: Summarization

Risk: The model adds details that are not found within the source material. Mitigation: Prompt the model with, “Summarize only from the text below. Do not add additional facts. Quote key claims and provide a citation.” Then utilize a verifier to ensure that every claim is supported by the source.

Example 2: Analytics Assistant

Risk: The model estimates numerical figures instead of calculating them accurately. Mitigation: Direct questions to a SQL generator for execution. Display both the natural language answer and the SQL code used, along with a link to the result.

Example 3: Customer Support

Risk: The model fabricates policy specifics or warranty terms. Mitigation: Implement RAG over your policy documents and require citations to specific paragraphs. If the pertinent section is absent, the assistant should indicate that it cannot provide an answer.

Example 4: Coding Assistant

Risk: The model recommends non-existent library functions. Mitigation: Incorporate a tool that inspects local environments or documentation. Execute code in a safe environment for examples and tests before delivering an answer.

Insights from Ongoing Research

Understanding and addressing hallucinations continues to be an active research domain. Several key themes are emerging from this exploration:

• Grounding is crucial: Utilizing retrieval and external tools has consistently shown improvements in factual accuracy for knowledge-intensive tasks [Lewis et al., 2020].
• Feedback is beneficial, but not exhaustive: Techniques such as RLHF and Constitutional AI enhance helpfulness and safety, but still gain from grounding and verification [Ouyang et al., 2022], [Bai et al., 2022].
• Models can gauge their uncertainty: Proper prompting or training allows models to express uncertainty, leading to improved abstention when they lack information [Kadavath et al., 2022].
• Evaluation processes need to be task-specific: No single metric captures every potential failure. Merging claim-level assessments, human evaluations, and adversarial tests yields a more reliable framework [Ji et al., 2023], [Min et al., 2023].

Limitations and Open Questions

Despite employing the most effective strategies, halting hallucinations completely remains elusive. Here are some pivotal open questions:

• Provable guarantees: How can we get close to formal assurances of correctness for complex tasks involving lengthy reasoning?
• Robustness to data distribution shifts: How do we maintain precision as facts evolve and fresh domains arise?
• Transparency in reasoning: How can we ensure that internal reasoning or explanations represent actual computations rather than post-hoc justifications?
• Scalable oversight: How do we supervise models executing tasks that are too intricate for exhaustive human validation?

Conclusion

Hallucinations are not a mysterious flaw; rather, they are an anticipated consequence of the training and operation of language models. The good news is, through grounding inputs, employing the right tools, focusing on careful decoding, and implementing verification, you can significantly reduce their occurrence. With a layered approach, achieving high reliability without compromising speed or user experience is entirely feasible.

By designing systems that verify claims, reference sources, and articulate uncertainty when uncertain, hallucinations can transform from common occurrences into rare anomalies.

FAQs

Do larger models experience fewer hallucinations?

Generally, yes—larger models tend to have greater knowledge and more effective calibration. However, size alone does not resolve hallucinations; grounding, tool access, and verification are still crucial.

Does decreasing temperature always mitigate hallucinations?

Lower temperatures typically decrease random errors by making outputs more deterministic. However, if a model is strongly incorrect due to data or prompt issues, a low temperature might simply result in consistent inaccuracies. It’s important to align temperature settings with grounding and verification efforts.

Can I compel a model to provide citations to prevent hallucinations?

Requesting sources can help, but without grounding, models can still fabricate references. Augment your system with retrieval mechanisms to ensure that citations come from verified documents and confirm the links.

Is chain-of-thought reasoning always superior?

Often, requiring structured reasoning or multiple passes is beneficial, but it does not guarantee accuracy. Pair it with verification and grounding, while also being cautious of privacy issues that may arise from sensitive data within reasoning paths.

What practices should I follow in regulated fields?

Mandate citations, utilize domain-specific knowledge bases, incorporate human review, and prefer tools over open-ended answers for critical calculations. Maintain audit logs and persistently evaluate with domain experts.

Sources

1. OpenAI. Why language models hallucinate.
2. Ji et al., 2023. Survey of hallucination in natural language generation.
3. Lin et al., 2022. TruthfulQA: Measuring how models imitate human falsehoods.
4. Holtzman et al., 2020. The Curious Case of Neural Text Degeneration.
5. Lewis et al., 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP.
6. Karpukhin et al., 2020. Dense Passage Retrieval for Open-Domain Question Answering.
7. Ouyang et al., 2022. Training language models to follow instructions with human feedback.
8. Bai et al., 2022. Constitutional AI: Harmlessness from AI Feedback.
9. Kadavath et al., 2022. Language Models (Mostly) Know What They Don’t Know.
10. Min et al., 2023. FactScore: Fine-grained Evaluation of Factuality in Long-form Text Generation.
11. OpenAI Evals: A framework for evaluating LLMs and systems.