In this newsletter we have selected two papers on how to use LLMs in empirical research:
The first by Jens Ludwig, Sendhil Mullainathan, and Ashesh Rambachan provides the theoretical framework – it establishes the fundamental econometric principles and validity conditions that economists must follow when using LLMs.
The second by Jing Cynthia Wu, Jin Xi, and Shihan Xie provides a concrete, successful implementation – it demonstrates exactly how to apply those principles in practice. It’s the perfect case study showing that when you follow the Ludwig framework properly (especially the validation requirement for estimation problems), LLMs can deliver impressive results.
Can we trust Large Language Models in empirical research?
Can economists use LLMs in empirical research without compromising rigor? The paper develops clear “rules of the road” that distinguish when LLM outputs can be trusted versus when they’ll mislead you.
Prediction Tasks
- Goal: Use text to predict economic outcomes (e.g., predicting stock returns from news or legislative outcomes from bill descriptions)
- The rule: Your sample must be outside the LLM’s training data. Otherwise the LLM simply memorizes the answer rather than actually predicting.
- What to do: Use open-source LLMs like Llama with publicly documented training dates. Never use closed models like GPT-4o where you can’t verify the training data.
- Evidence: When the authors asked GPT-4o to complete Congressional bills from partial text, it perfectly reproduced over 300 bills—clear memorization, not genuine prediction.
Estimation Tasks
- Goal: Use LLM outputs to automatically measure economic concepts (e.g., classifying policy topics, extracting sentiment) for downstream analysis
- The rule: You must collect validation data from your existing measurement approach. Small measurement errors in the LLM propagate into large errors in your final estimates.
- What to do:
1. Have human coders label a subset of your data the traditional way
2. Compare LLM labels to human labels to identify errors
3. Statistically correct for these errors in your final results
4. You can use even small validation samples (2.5%) and still get reliable estimates - Evidence: Different prompts to the same LLM produced labels that looked equally accurate, yet led to regression coefficients that differed by 2-3 times, sometimes even flipping signs. Without validation data, you’d have no way to know which answer was reliable.
The key finding
LLMs are brittleness machines: Tiny changes in how you ask the question (different prompts, different models) produce wildly different downstream results. But this brittleness is manageable with validation data and bias correction.
Monte Carlo simulations showed that when researchers followed the framework (especially the validation step), bias-corrected estimates eliminated errors and restored proper statistical coverage—often achieving better precision than traditional approaches.
Practical bottom line
- For prediction: Use time-stamped, open-source LLMs and ensure your data post-dates their training cutoff.
- For estimation: Always collect benchmark validation data from a subset of your sample, then apply statistical bias correction.
- The promise: When you follow these rules, LLMs let you expand what empirical research can accomplish—automating expensive measurement tasks and unlocking insights from text data.
- The warning: Ignore these rules and you’ll get confidently-stated nonsense that passes all your initial checks.
Large Language Models: An Applied Econometric Framework
Authors: Jens Ludwig, Sendhil Mullainathan, Ashesh Rambachan
From: University of Chicago, MIT
How much cheaper can economics surveys get?
In this paper, Jing Cynthia Wu, Jin Xi, and Shihan Xie demonstrate that LLMs can conduct sophisticated economic surveys at 1/1000th the cost of traditional online surveys while enabling research designs impossible with human respondents.
Use LLM personas as synthetic survey respondents instead of paying real people. Cost: 1/1000th of traditional surveys. Benefit: replicate human survey results while enabling new research designs.
How it works
- Date-restricted prompts: Tell the LLM to answer as if it’s a specific date in the past (e.g., “It’s January 2020, what’s your inflation forecast?”)
- Persistent personas: Use identical demographic profiles across multiple survey waves
- Validation: Compare LLM responses to actual human survey data to verify accuracy
Findings
The authors replicated a major 80,000-person human inflation expectations survey using just 200 LLM personas. The LLM survey reproduced the key finding: households respond less to new information when inflation is already high.
Cost per response: Humans $1-5 → LLMs $0.005
Beyond replication, LLMs enable:
- 35-year historical coverage (expanding 10-year human survey back to 1990)
- Identifying economic reasoning (e.g., discovering that high-inflation respondents emphasize mean-reversion)
- Monthly tracking of how beliefs change post-treatment
- Clean experimental identification using information not yet publicly available
LLM Survey Framework: Coverage, Reasoning, Dynamics, Identification
Authors: Jing Cynthia Wu, Jin Xi, Shihan Xie
From: University of Illinois Urbana-Champaign, Chinese Academy of Sciences
