GPT-4 Makes Up Facts 3% of the Time Smaller Models: Up to 15%
Hallucination rates vary wildly by model size and task type. Enterprises are quietly absorbing billions in correction costs. The reliability curve nobody talks about.
3%GPT-4 hallucination rate on factual queries
8–15%Smaller model hallucination rates
$3.1BEstimated annual enterprise cost (US) from AI errors
62%Enterprises who've had AI output errors cause real problems
Choose your depth. The data doesn't change — just the explanation.
AI sometimes makes up information that sounds true but isn't — this is called "hallucination." Big, expensive AI models do this about 3% of the time. Cheaper, smaller models do it 8-15% of the time. That means if you asked a small AI 100 questions, it would make up wrong answers 8 to 15 times. For businesses using AI to answer thousands of questions per day, that adds up to a lot of mistakes.
Hallucination refers to when AI models generate factually incorrect statements with full confidence. The rate depends on model size (parameter count), task type, and how "out of distribution" the question is from training data. GPT-4 maintains a ~3% error rate on well-benchmarked factual tasks. But smaller models — the ones companies use to cut API costs — run 8-15%. In legal, medical, or financial contexts, even 3% is catastrophically high. Enterprises are building human-in-the-loop verification systems that eat back most of the efficiency gains.
Hallucination benchmarks: TruthfulQA (Lin et al., 2022) measures propensity to endorse false statements; HELM (Liang et al., 2022) measures factual accuracy across 42 scenarios. GPT-4 TruthfulQA score: ~59% (improved but not reliable). The $3.1B enterprise cost estimate uses: number of US enterprises using AI (Gartner 2023: 55% of large enterprises) × average false output rate × cost per manual review ($0.35-$2.50/query depending on domain). Legal hallucination costs are highest: one fabricated case citation costs $10K-$100K in review/malpractice risk.
Benchmarks: TruthfulQA (lin2022truthfulqa), HaluEval (li2023halueval), HELM (liang2022holistic). Model-specific rates from: Scale AI SEAL leaderboard, LMSYS Chatbot Arena. Enterprise cost model: Gartner Magic Quadrant AI survey 2023 (55% adoption large enterprise) × average query volume × error rate × review cost. Legal cost per error: American Bar Association AI task force estimates, 2023. Key API: OpenAI Evals framework (github.com/openai/evals) for custom hallucination measurement.
Hallucination Rate by Model
Not all AI models hallucinate equally. Larger, more expensive models are more reliable — but even the best models make costly errors at scale.
Factual Accuracy by Model (Factual Query Benchmark)
TruthfulQA and HaluEval benchmarks, 2024. Lower hallucination rate = better.
Claude 3 Opus
2.5%
Best-in-class factual accuracy
GPT-4o
3.0%
Strong general knowledge
Gemini Ultra
3.8%
Strong on Google-indexed facts
GPT-3.5 Turbo
8.2%
Popular cost-saving choice
Mistral-7B
11.9%
Open-source, cost-effective
Llama-2-7B
14.7%
High rate, widely deployed
Falcon-7B
16.3%
Older architecture
GPT-3 (2020)
22.0%
Historical baseline
Task Type Matters as Much as Model
Hallucination rates spike on tasks requiring specific dates, statistics, citations, and rare knowledge — precisely the tasks enterprises use AI for most.
Hallucination Rate by Task Type (GPT-4)
Even the best models fail on specific task types.
Enterprise Cost by Sector (Annual, $M)
Estimated cost from AI errors requiring correction.
The Lawyer Problem
In 2023, attorneys Mata and LoDuca submitted a brief to a federal court citing six AI-generated case citations that didn't exist. The judge fined them $5,000 each. This isn't isolated — it's systemic. GPT-4's 3% error rate × thousands of legal queries = routine fictional citations.
Sources
• Lin, S., Hilton, J., Evans, O. (2022). "TruthfulQA: Measuring How Models Mimic Human Falsehoods." ACL 2022.
• Li, J. et al. (2023). "HaluEval: A Large-Scale Hallucination Evaluation Benchmark." EMNLP 2023.
• Liang, P. et al. (2022). "Holistic Evaluation of Language Models (HELM)." NeurIPS 2022.
• Gartner. "AI Adoption in the Enterprise." 2023 Survey Report.
• Mata v. Avianca, Inc. (2023). S.D.N.Y. Order on AI-generated fictitious citations.