Key Scientific Paper Redux: Executives vs Chatbots

This week we provide another redux for a key scientific paper, “Executives vs Chatbots: Unmasking Insights Through Human-AI Differences in Earnings Conference Q&A.”[i] In particular, the researchers used Large Language Models (LLM) in a very clever way. Namely, they asked LLM to answer questions about a company’s performance and then contrasted the content of answers directly from management to the answers of the LLMs.

Their key assumption is that the LLM’s answers would be made up of already known/likely already priced information about the company. By contrast, the portions of the answers from management not captured by the LLM would be considered to be new and unique information that deserved greater scrutiny by investors. This approach allowed them to create a new measure which they refer to as the Human AI Difference, or HAID.

At Deception And Truth Analysis we also believe that some of the researchers’ key findings support our own beliefs about the content shared by management on earnings calls. We have found, for example, that the average DATA Score of earnings calls is 29.7% versus an average DATA Score of 8.1% for quarterly and annual reports. In other words, earnings calls are 3.67x more truthful. Why would this be?

We have hypothesized that management carefully manages the narratives of earnings calls and that they conduct a form of theater to ensure that the very best light is shown on company performance. Good information is readily shared, whereas bad information is either omitted or has spin applied so as to influence investor perceptions.

Fascinatingly, the researchers in this study find that their HAID score is, in fact, positively correlated with higher future earnings, as well as a higher propensity to issue future earnings guidance. In other words, the degree of new information shared by management is directly proportional to good future outcomes of the company. In short, management does manipulate the investment community when disclosing information on earnings calls.

The steps to creating the Human AI Difference (HAID) are straightforward:

1. They use three LLMs – ChatGPT, Google Bard, and a freeware LLM – to generate responses to investor questions based on existing information related to macroeconomics, industry trends, and firm conditions. Since LLMs try to provide optimized responses to user prompts, they hypothesize that these responses are a close approximation to the expected or already known and priced information about a company.
2. By examining the semantic differences between management teams’ answers and those provided by the LLM, they believe that the quantifiable differences in semantics serve as a proxy for the informativeness of the Q&A section of earnings calls. This semantic similarity percentage was then subtracted from 1 in order to generate the HAID. For example, if the semantic similarity between the answers provided by the LLMs and management was high, say 85%, then the HAID would be 1 – 0.85 = 0.15. This would indicate a lower new information content provided in the Q&A than if the similarity had been, say 45% where the HAID would be 1 – 0.45 = 0.55.
3. One of the key problems identified in using LLMs is that they can invent information or answers that are complete fabrications. These fabrications are known as hallucinations. In order to minimize this problem, the researchers provided the LLMs with the same information that investors listening on the call would have at their disposal prior to participating in the call. For example, the details provided by companies in their earnings press releases.
4. In total the researchers examined 190,538 earnings calls, as well as stock price information before and after the calls (from CRSP), as well as earnings estimates (from I/B/E/S). Along with this information the researchers used other financial data about the company and other measures. All of this led to a final sample of 104,932 earnings calls between 2004 and 2020 and corresponded to 5,570 unique companies.
5. The researchers hypothesized that a higher HAID would capture more informational content and lead to more-informed trading activity, reduce information asymmetries, and lead to greater market liquidity. The market action was summarized using absolute cumulative abnormal stock returns and abnormal trading volumes during and after the earnings calls, as well as the number of analyst forecast revisions and the accuracy of their estimates, too.
6. To ensure that other key performance metrics were not the source of the pricing and volume actions they controlled for other business metrics, such as firm size, market-to-book ratios, leverage, R&D, ROA, stock price volatility, analyst coverage, special items, and institutional investor ownership.
7. Another hypothesis considered is that a higher HAID is more likely to convey good news rather than bad news as measured by higher future earnings growth.
8. Last, the researchers also hypothesized that a higher HAID ratio is also likely more associated with earnings guidance on the part of management. This rests on an assumption that when management expects good news they are more likely to want to share that guidance on earnings calls.

1. HAID is strongly positively associated with the absolute cumulative abnormal return and abnormal trading volume in the two-days trading window before and after the conference call date. Specifically, a move in HAID from the 10th to 90th percentiles is associated with a 0.143 higher abnormal trading volume and a 2.6% higher absolute cumulative abnormal return.
2. They found that a higher HAID results in smaller analyst forecast errors of 11.1%. They also found less analyst forecast dispersion of 4.7%. Dispersion here is measured as the level of disagreement among analysts’ ratings.
3. Regarding liquidity, the researchers found that HAID statistically and significantly negatively correlated with the bid-ask spread, as well as the Amihud Ratio. Specifically, an increase in HAID from the 10th to 90thpercentile resulted in a 3.8% lower bid-ask spread and a 5.4% lower Amihud Ratio.
4. That earnings calls contain more good news than bad news is supported by the fact that a higher HAID ratio is associated with higher future earnings growth of 1.3%.
5. An increase in HAID also results in a 15.2% increase in the probability that management issues quantitative earnings guidance.

This study’s researchers find that the use of LLMs, as well as the use of traditional semantic similarity measures in Natural Language Processing provide new and unique information for investors. Additionally, they find that the amount of new information disclosed by management is predictive of multiple future outcomes including: higher abnormal returns, higher abnormal trading volumes, smaller forecast errors by analysts, less variability in all analysts’ forecasts, smaller bid-ask spreads, and higher liquidity. Additionally, and most interesting to DATA, the researchers found that a relatively higher ratio of new information is associated with future good outcomes at companies. In other words, this is evidence that when management discloses new information it is typically good news, not bad news.

Regarding the HAID ratio being hypothesized to contain more information about good news rather than bad: “This conjecture is driven by the incentives of managers to elaborate on positive developments and disclose private information, while being motivated to conceal negative news.”

The DATA Score for this article is 93.91%, or 99.99th %-ile truthful, or very low risk.  

[i]Bai, John (Jianqiu), Nicole Boyson, Yi Cao, Miao Liu, and Chi Wan. “Executives vs Chatbots: Unmasking Insights Through Human-AI Differences in Earnings Conference Q&A.” SSRN Posted 22 Jun 2023 https://papers.ssrn.com/sol3/papers.cfm?abstract_id=4480056