Advancing central bank natural language models

The role of text mining in modern central banking

Bricongne et al. provide a comprehensive overview of how text mining techniques are being utilized by central banks and supervisory institutions. The authors find that text mining has become an increasingly important tool for central banks to gauge economic conditions, assess risks, and improve communication.

• Economic outlook and inflation:
  – Text mining helps in nowcasting and forecasting economic conditions, often providing more timely information than traditional indicators.
  – It is particularly useful for measuring inflation expectations and sentiment.
• Financial stability:
  – Text-based sentiment analysis can predict market returns, volatility, and financial stress.
  – It’s effective in monitoring risks in housing markets, which are prone to sentiment-driven fluctuations.
• Supervision of financial institutions:
  – Text mining can assess risks to individual banks by analyzing annual reports, CEO letters, and social media sentiment.
  – It helps in evaluating compliance with regulations, such as ESG disclosures.
• Climate change:
  – Central banks are using text analysis to assess climate-related risks and their impact on financial stability.
  – It helps in measuring climate uncertainty and its effects on markets.
• Central bank communication:
  – Text analysis of central bank communications helps in understanding their impact on financial markets and public expectations.
  – It provides feedback on how to best convey decisions and policies.
• Complementary to traditional methods:
  – While powerful, text mining is seen as a complement to, rather than a replacement for, traditional indicators and procedures.
  – Combining text-based and traditional indicators often yields the best results.
• Future prospects:
  – Generative AI and Large Language Models (LLMs) are opening new frontiers for the use of textual data in central banking.
  – These tools show promise in areas like forecasting, policy analysis, and communication.
• Challenges:
  There are challenges associated with text mining, these include interpretative issues, ambiguity in language, and the need for domain-specific knowledge.

Should Central Banks Care About Text Mining?
Authors: Jean-Charles Bricongne, Raquel Caldeira, Baptiste Meunier
From: Banque de France

Evaluating language models for monetary policy analysis

(This paper presupposes some knowledge of LLMs)
Gambacorta et al., in this paper, introduce and evaluate Central Bank Language Models (CB-LMs), which are domain-adapted language models specifically trained for central banking tasks. Here are the key findings:

• Development of CB-LMs:
  – CB-LMs were created by retraining foundational models like BERT and RoBERTa on a large corpus of central bank speeches, policy documents, and research papers.
  – Six unique CB-LMs were developed using different combinations of foundation models and training datasets.
• Performance in predicting central banking idioms:
  – CB-LMs outperformed foundation models in a masked word test using 100 common central banking idioms.
  – CB-LMs correctly predicted 90 out of 100 masked words, compared to 60 for RoBERTa and 53 for BERT.
• Monetary policy sentiment analysis:
  – RoBERTa-based CB-LMs showed improved performance over the foundational RoBERTa model in classifying monetary policy stance from FOMC statements.
  – The best-performing CB-LM achieved a mean accuracy of about 84%, compared to 81% for the foundational RoBERTa model.
• Comparison with state-of-the-art generative LLMs:
  – In simpler tasks like sentence-level sentiment classification, many generative LLMs did not outperform smaller encoder-only models like BERT and RoBERTa.
  – However, in more complex scenarios with limited training data and longer texts, advanced generative LLMs like ChatGPT-4 and Llama-3 70B outperformed CB-LMs.
• Performance in complex tasks:
  – In a challenging task of classifying monetary policy sentiment in longer news texts with limited training data, ChatGPT-4 and Llama-3 70B achieved superior performance (80-81% accuracy) compared to the best CB-LMs (65% accuracy).
• Considerations for using generative LLMs in central banking:
  – The authors discuss challenges related to confidentiality, privacy, transparency, replicability, cost-efficiency, and infrastructure requirements when using proprietary or open-source generative LLMs in central banking contexts.

Gambacorta et al. advocate for strategic integration of these models into central banking analytical frameworks and encourages knowledge sharing among central banks to advance the use of language models in monetary policy decision-making.

CB-LMs: language models for central banking
Authors: Leonardo Gambacorta, Byeungchun Kwon, Taejin Park, Pietro Patelli, Sonya Zhu
From: BIS