Modelling bounded rationality using deep reinforcement learning
In this paper, Chen et al. show that deep reinforcement learning can be used by economists to solve complex behavioral problems. Their approach consists of using deep reinforcement learning to combine a standard dynamic stochastic general equilibrium model with flexible expectations formation. In particular, they are replacing bounded rationality agents with artificially “intelligent agents” who learn and make optimal decisions through repeated experience.
Agents maximize some long-term reward, similar to the cumulated discounted sum of future utilities as in classical macroeconomic models, by interacting with a generally unknown environment, namely the model economy. Agents obtain observations of state variables from this environment and then they decide how much to consume this period. They are then rewarded by the environment, which also returns a new state observation, and so on.
[T]he deep reinforcement learning agent recovers all possible steady state solutions under the different policy regimes. Given that this learning behaviour is more general, as seen through the lens of generalised policy iteration, this widens the scope of solutions of interest, irrespective of their local dynamics. This is achieved by the “global” nature of deep reinforcement learning without the need of a local model approximation. Agents’ learning and behaviour is ultimately determined by the long-run reward subject to constraints. In our case this consists of household utility subject to the standard inter-temporal accounting constraint.
Deep Reinforcement Learning in a Monetary Model
Authors: Mingli Chen, Andreas Joseph, Michael Kumhof, Xinlei Pan, Rui Shi, Xuan Zhou
From: University of Warwick, Bank of England, University of California, Berkeley, Deakin University
Estimating nonlinear HANK models using neural networks
In this paper, Kase et al. suggest using machine learning to estimate and solve models without resorting to approximations or simplifications that necessarily distort their properties and predictions. They apply their method to estimate a quantitative Heterogeneous Agent New Keynesian (HANK) model in its nonlinear specification. They show that their method is capable of simultaneously dealing with a large number of state variables and parameters, nonlinear dynamics, heterogeneity as well as aggregate uncertainty.
A critical step we propose in this paper is to use the model’s parameter vector as direct input in the neural network. In other words, we treat the parameters as pseudo state variables by exploiting the scalability of neural networks. We then train this extended neural network with the parameters as inputs over the entire parameter space. This step turns out to be critical because it yields the model’s solution mapping (i.e., the mapping from the parameter space to the model’s equilibrium law of motion) instead of just one point in this solution mapping – associated with the model solution for a single point in the parameter space.
Estimating Nonlinear Heterogeneous Agents Models with Neural Networks
Authors: Hanno Kase, Leonardo Melosi, Matthias Rottner
From: University of Minnesota, Federal Reserve Bank of Chicago, Deutsche Bundesbank
