{"id":14334,"date":"2023-11-28T15:28:26","date_gmt":"2023-11-28T20:28:26","guid":{"rendered":"https:\/\/jasonapollovoss.com\/web\/?p=14334"},"modified":"2025-09-05T16:25:39","modified_gmt":"2025-09-05T22:25:39","slug":"key-scientific-paper-redux-can-ai-read-the-minds-of-corp-execs","status":"publish","type":"post","link":"https:\/\/jasonapollovoss.com\/web\/2023\/11\/28\/key-scientific-paper-redux-can-ai-read-the-minds-of-corp-execs\/","title":{"rendered":"Key Scientific Paper Redux: Can AI Read the Minds of Corp Execs?"},"content":{"rendered":"

[et_pb_section fb_built=”1″ admin_label=”section” _builder_version=”4.16″ global_colors_info=”{}” theme_builder_area=”post_content”][et_pb_row admin_label=”row” _builder_version=”4.16″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” global_colors_info=”{}” theme_builder_area=”post_content”][et_pb_column type=”4_4″ _builder_version=”4.16″ custom_padding=”|||” global_colors_info=”{}” custom_padding__hover=”|||” theme_builder_area=”post_content”][et_pb_text admin_label=”Text” _builder_version=”4.16″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” global_colors_info=”{}” theme_builder_area=”post_content”]<\/p>\n

\n
<\/div>\n<\/figure>\n

In this Key Scientific Paper Redux of \u201cCan AI Read the Minds of Corporate Executives?,\u201d[i]<\/a>\u00a0we summarize the findings of researchers who were interested in whether or not using Natural Language Processing may be used to assess disclosures from companies in their regulatory filings that would allow them to predict future earnings surprises in subsequent quarters. The answer to that question is: yes. This finding comports with\u00a0our own back-test analyses<\/a>\u00a0that finds that there is still signal contained in within quarterly and annual reports over a year later.<\/span><\/p>\n

\n

<\/strong><\/span><\/h4>\n

Study Details<\/strong><\/span><\/h3>\n<\/div>\n

The researchers considered three different high-level approaches to extract meaning from the words contained in company 10-Ks and 10-Qs from 1993 thru 2021 and had them compete against one another to predict future earnings surprises. Specifically, those methods are:<\/span><\/p>\n

1.\u00a0Method 1: Sentiment<\/u>.<\/span><\/p>\n

\u00a0\u00a0\u00a0a. A keyword sentiment lexicon developed by the researchers Loughran and McDonald in 2011 that had human-based researchers encoding words in financial reports as to their meaning.<\/span><\/p>\n

\u00a0\u00a0\u00a0b. A tweaking of Google\u2019s pre-trained Large Language Model BERT (Bidirectional Encoder Representations from Transformers) known as FinBERT, created by Huang, et al. in 2022.<\/span><\/p>\n

\u00a0\u00a0\u00a0c. Simply looking at the length of the MD&A and Risk Factors sections.<\/span><\/p>\n

2.\u00a0Method 2: Bag-of-Words<\/u>.<\/span><\/p>\n

\u00a0\u00a0\u00a0a. A manual word classification scheme along with a regression model similar to that employed by Jegadeesh and Wu in 2013. Here the meanings of words are classified and then given weights by a regression model.<\/span><\/p>\n

\u00a0\u00a0\u00a0b. The same type of approach as above, but as suggested by a difference scheme proposed by Manela and Moreira in 2017.<\/span><\/p>\n

3.\u00a0Method 3: Hierarchical Transformer Approach LLM<\/u>. The method proposed by the authors of the paper who use a Large Language Model in a novel way. Specifically, they focus on:<\/span><\/p>\n

\u00a0\u00a0\u00a0a. The MD&A and Risk Factors sections of reports.<\/span><\/p>\n

\u00a0\u00a0\u00a0b. Given that current quarterly earnings announcements are noisy, they train their machine learning algorithms on next quarter\u2019s earnings announcement surprises.<\/span><\/p>\n

\n

<\/strong><\/span><\/h4>\n

Major Findings<\/strong><\/span><\/h3>\n<\/div>\n

For each of the Methods\u2019 performance summarized below the authors\u2019 criteria was to rank stocks based on their earnings surprise forecasts into quintiles and then evaluate the out of sample performance of the High-minus-Low strategy that buys the highest quintile category and sells the lowest quintile category. Furthermore, for these quintile portfolios they looked at both the equal weighted (EW) and value weighted (VW) portfolio returns. Last, they controlled for additional factors that might skew their results, such as cross-sectional and time-series regressions with the time horizon and various firm characteristics.\u00a0<\/span><\/p>\n

For Method 2 the researchers used various statistical and machine learning methods to develop weightings for the words identified as predictive. Which statistical method was used is shown in the parentheses, below.<\/span><\/p>\n

Below we do not describe the full voluminous output of the researchers. But the authors also looked at CAPM returns, as well as Fama-French 5-factor and 6-factor returns. For those interested in these measures of performance, please consult\u00a0the paper<\/a>\u00a0which is publicly available.<\/span><\/p>\n

1. Method 1 performance is as follows:<\/span><\/p>\n

\u00a0\u00a0\u00a0a. Keyword sentiment lexicon: EW = -2.1%; VW = +4.5%.<\/span><\/p>\n

\u00a0\u00a0\u00a0b. FinBERT: EW = +15.0%; VW = +31.3%.<\/span><\/p>\n

\u00a0\u00a0\u00a0c. Length of MD&A disclosures: EW = -18.9%; VW = -26.4% (in other words, shorter disclosures perform better).<\/span><\/p>\n

\u00a0\u00a0\u00a0d. Length of Risk Factors disclosures: EW = +18.6%; VW = -8.9% (in other words, shorter disclosures perform better).<\/span><\/p>\n

2. Method 2 performance is as follows:<\/span><\/p>\n

\u00a0\u00a0\u00a0a. Bag-of-Words classification, version 1 (Ordinary Least Squares): EW = +22.4%; VW = +3.9%.<\/span><\/p>\n

\u00a0\u00a0\u00a0b. Bag-of-Words classification, version 2 (Loughran & McDonald OLS): EW = +18.7%; VW = +21.9%.<\/span><\/p>\n

\u00a0\u00a0\u00a0c. Bag-of-Words classification, version 3 (Elastic Nets): EW = +33.3%; VW = +18.1%.<\/span><\/p>\n

\u00a0\u00a0\u00a0d. Bag-of-Words classification, version 4 (Lasso): EW = +22.0%; VW = +10.3%.<\/span><\/p>\n

\u00a0\u00a0\u00a0e. Bag-of-Words classification, version 5 (Support Vector Regression): EW = +40.0%; VW = +41.8%.<\/span><\/p>\n

\u00a0\u00a0\u00a0f. Bag-of-Words classification, version 6 (XGBoost): EW = +25.6%; VW = +39.7%.<\/span><\/p>\n

\u00a0\u00a0\u00a0g. Bag-of-Words classification, version 7 (Random Forest): EW = -4.4%; VW = -19.1%.<\/span><\/p>\n

\u00a0\u00a0\u00a0h. Bag-of-Words classification, version 8 (Feed Forward Neural Networks): EW = +18.7%; VW = +21.7%.<\/span><\/p>\n

3. Method 3 performance is as follows:<\/span><\/p>\n

\u00a0\u00a0\u00a0a. Frozen BERT: EW = +40.7%; VW = +43.0%.<\/span><\/p>\n

\u00a0\u00a0\u00a0b. FtBERT: EW = +43.9%; VW = +56.1%.<\/span><\/p>\n

\n

<\/strong><\/span><\/h4>\n

Conclusions<\/strong><\/span><\/h3>\n<\/div>\n

Among the findings of the researchers are that traditional Natural Language Processing techniques are not able to identify future positive or negative changes in firms\u2019 valuations. Next, off-the-shelf Large Language Models (LLMs), even those trained on financial targets, while good at predicting future earnings surprises, are not better than simpler estimators such as the lengths of companies\u2019 MD&A and Risk Factors sections of their quarterly and annual reports. Third, fine-tuning and training LLMs is a solution. The researchers\u2019 offering, FtBERT provides superior results in identifying future positive or negative changes in earnings. Last, the researchers find that there is robust unpriced information contained in both quarterly and annual reports.<\/span><\/p>\n

<\/strong><\/span><\/p>\n

Quotes of Note<\/strong><\/span><\/h3>\n