Future Is Unevenly Distributed: Forecasting Ability of LLMs Depends on What We're Asking
Paras Chopra
Abstract
Forecasting performance of Large Language Models varies significantly across different domains and question types, influenced by context and external knowledge.
Large Language Models (LLMs) demonstrate partial forecasting competence across social, political, and economic events. Yet, their predictive ability varies sharply with domain structure and prompt framing. We investigate how forecasting performance varies with different model families on real-world questions about events that happened beyond the model cutoff date. We analyze how context, question type, and external knowledge affect accuracy and calibration, and how adding factual news context modifies belief formation and failure modes. Our results show that forecasting ability is highly variable as it depends on what, and how, we ask.
Community
LLMs forecasting ability on real world questions from prediction markets (such as polymarket) varies significantly by category.
While addition of news helps, it also adds certain failure modes such as definition drift, recency bias and rumour anchoring
This is an automated message from the Librarian Bot. I found the following papers similar to this paper.
The following papers were recommended by the Semantic Scholar API
- Assessing Large Language Models in Updating Their Forecasts with New Information (2025)
- LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arena (2025)
- Can Large Language Models Express Uncertainty Like Human? (2025)
- RETuning: Upgrading Inference-Time Scaling for Stock Movement Prediction with Large Language Models (2025)
- AIA Forecaster: Technical Report (2025)
- How AI Forecasts AI Jobs: Benchmarking LLM Predictions of Labor Market Changes (2025)
- Incoherent Beliefs&Inconsistent Actions in Large Language Models (2025)
Please give a thumbs up to this comment if you found it helpful!
If you want recommendations for any Paper on Hugging Face checkout this Space
You can directly ask Librarian Bot for paper recommendations by tagging it in a comment:
@librarian-bot
recommend
Models citing this paper 0
No model linking this paper
Datasets citing this paper 0
No dataset linking this paper
Spaces citing this paper 0
No Space linking this paper
Collections including this paper 0
No Collection including this paper