An insightful look at the new AI tools

Do the New AI Tools Actually Reason?

In this paper, Perez-Cruz and Shin present important conclusions about the reasoning capabilities of large language models (LLMs), particularly in the context of logical puzzles. The key takeaways from the paper are:

• Distinctive Pattern of Failure: When faced with a logical puzzle that requires reasoning about the knowledge of others and counterfactuals, LLMs display a distinctive pattern of failure. They perform well with the original wording of the puzzle but poorly when incidental details are changed, indicating a lack of true understanding of the underlying logic.
• Cheryl’s Birthday Puzzle: The paper presents the well-known Cheryl’s birthday puzzle and discusses how LLMs performed when presented with the original and modified versions:

The contrast between the flawless logic when faced with the original wording and the poor performance when faced with incidental changes in wording is very striking. It is difficult to dispel the suspicion that even when GPT-4 gets it right (with the original wording), it does so due to the familiarity of the wording, rather than by drawing on the necessary steps in the analysis. In this respect, the apparent mastery of the logic appears to be superficial.

While LLMs have demonstrated impressive capabilities, the findings suggest that caution should be exercised in deploying LLMs in contexts that demand rigorous reasoning, especially in economic analysis.

Testing the cognitive limits of large language models
Authors: Fernando Perez-Cruz and Hyun Song Shin
From: BIS

Who benefits from the deployment of generative AI

In this paper, Brynjolfsson et al. study the adoption of a generative AI tool that provides conversational guidance for customer support agents. The authors look at the deployment of a chat assistant using data from 5,000 agents working for a Fortune 500 software firm that provides business process software. The tool they study is built on a recent version of the Generative Pre-trained Transformer (GPT) family of large language models developed by OpenAI.

• Impact on Productivity: The deployment of AI assistance in customer service leads to a 14 percent increase in productivity as measured by the number of customer issues they are able to resolve per hour. The study found a small increase in chat resolution rates, a decrease in average handle time, and an increase in chats per hour, indicating improved productivity without compromising problem-solving or customer satisfaction.
• Impacts by Agent Skill and Tenure: The authors also discuss the implications of AI deployment based on agent skill and tenure. They suggest that high-skill workers play a crucial role in model development but may see smaller direct benefits in terms of improving their own productivity.  In particular, the authors argue that high-skill workers may have less to gain from AI assistance precisely because AI recommendations capture the potentially tacit knowledge embodied in their own behaviors.

In contrast to studies of prior waves of computerization, we find that these gains accrue disproportionately to less-experienced and lower-skill workers. We argue that this occurs because ML systems work by capturing and disseminating the patterns of behavior that characterize the most productive agents.

Generative AI at Work
Authors: Erik Brynjolfsson, Danielle Li, Lindsey Raymond
From: Stanford University, MIT