Large Language Models in econometrics
This paper by Jens Ludwig, Sendhil Mullainathan, and Ashesh Rambachan explores how large language models (LLMs) can be effectively used in empirical research while addressing their limitations. Here is a summary of the key findings:
-
Prediction Problems: For LLMs to be validly used in prediction tasks, such as predicting economic variables from text, there must be no “training leakage” between the LLM’s training dataset and the researcher’s sample. This can be ensured by using open-source LLMs with documented training data. Open-source models like Llama or StoriesLM can provide advantages in prediction tasks by leveraging their pre-trained knowledge on large text corpuses.
-
Estimation Problems: When using LLMs to automate the measurement of economic concepts (e.g., sentiment analysis), researchers must collect validation data on a subset of their sample using their preferred measurement procedure. This approach preserves econometric guarantees like consistency and asymptotic normality while reducing data collection costs. The LLM can improve precision by acting as a missing-value imputer, leveraging its pre-trained knowledge.
-
Importance of Benchmark Data: Without benchmark data, using LLM outputs for estimation is risky unless the LLM’s outputs are assumed to be error-free, which is unlikely given the documented brittleness of LLMs. Collecting benchmark data is crucial for de-biasing LLM outputs and ensuring reliable empirical estimates.
-
Applications and Challenges: The paper illustrates these principles with applications in finance and political economy, highlighting the stringent requirements for reliable use of LLMs. It also notes that violating these conditions can lead to unreliable estimates, underscoring the need for careful methodology when integrating LLMs into empirical research.
Large Language Models: An Applied Econometric Framework
Authors: Jens Ludwig, Sendhil Mullainathan, Ashesh Rambachan
From: University of Chicago, MIT
Leveraging Large Language Models for theory development
This paper by Matteo Tranchero, Cecil-Francis Brenninkmeijer, Arul Murugan, and Abhishek Nagaraj discusses the use of Large Language Models (LLMs) in management research, particularly through a framework called Generative AI-Based Experimentation (GABE). GABE allows researchers to simulate complex strategic interactions using AI agents, which can mimic human behavior in organizational settings. The authors apply GABE to study the “streetlight effect,” a phenomenon where people tend to search where data is readily available, i.e. where it is easiest to look, often to their detriment.
Key findings:
- Replication of human experiments: The study successfully replicates an online lab experiment using AI agents, showing that they behave similarly to human subjects. Both humans and AI agents tend to earn less when shown the location of a medium-value option, as they herd around this safer choice instead of exploring for higher-value options.
- Extensions and mechanisms: By varying experimental setups, relaxing theoretical assumptions, and incorporating heterogeneous agent preferences, the study uncovers new mechanisms and boundary conditions for the streetlight effect. For example, introducing payoff rivalry eliminates the streetlight effect, as agents avoid splitting rewards. Social conformity is identified as a mechanism driving herding behavior.
- Heterogeneous preferences: Introducing risk-loving or pro-social agents increases exploration and reduces the streetlight effect. This suggests that even a small proportion of agents with alternative preferences can significantly impact group behavior.
- Comparison with direct elicitation: The study finds that using AI agents in GABE provides more accurate and nuanced insights into mechanisms compared to directly asking LLMs to predict outcomes.
Theorizing with Large Language Models
Authors: Matteo Tranchero, Cecil-Francis Brenninkmeijer, Arul Murugan, Abhishek Nagaraj
From: University of Pennsylvania, University of California, Berkeley
