{"id":14267,"date":"2023-02-28T14:09:47","date_gmt":"2023-02-28T19:09:47","guid":{"rendered":"https:\/\/jasonapollovoss.com\/web\/?p=14267"},"modified":"2025-09-05T15:05:46","modified_gmt":"2025-09-05T21:05:46","slug":"key-scientific-paper-redux-how-humans-impair-automated-deception-detection-performance","status":"publish","type":"post","link":"https:\/\/jasonapollovoss.com\/web\/2023\/02\/28\/key-scientific-paper-redux-how-humans-impair-automated-deception-detection-performance\/","title":{"rendered":"Key Scientific Paper Redux: How humans impair automated deception detection performance"},"content":{"rendered":"

[et_pb_section fb_built=”1″ admin_label=”section” _builder_version=”4.16″ global_colors_info=”{}” theme_builder_area=”post_content”][et_pb_row admin_label=”row” _builder_version=”4.16″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” global_colors_info=”{}” theme_builder_area=”post_content”][et_pb_column type=”4_4″ _builder_version=”4.16″ custom_padding=”|||” global_colors_info=”{}” custom_padding__hover=”|||” theme_builder_area=”post_content”][et_pb_text admin_label=”Text” _builder_version=”4.16″ background_size=”initial” background_position=”top_left” background_repeat=”repeat” global_colors_info=”{}” theme_builder_area=”post_content”]<\/p>\n

\n
<\/div>\n<\/figure>\n

At Deception And Truth Analysis (D.A.T.A.) we use Natural Language Processing (NLP) to look for the known behavioral differences between deceivers and truth-tellers in communications. Frequently we encounter an objection from our prospects that they already are excellent at uncovering deception in documents. Importantly, the findings of \u201cHow humans impair automated deception detection performance\u201d[i]<\/a>calls that claim into question scientifically. Understanding this key scientific paper\u2019s findings is important for due-diligence pros reliant on the representations of people in their work; i.e. most of us.<\/span><\/p>\n

\n

<\/strong><\/em><\/span><\/h4>\n

How humans impair automated deception detection performance<\/strong><\/em>: Study Details<\/strong><\/span><\/h3>\n<\/div>\n

For decades the academic community has researched how to better identify if someone is lying or telling the truth. More recently researchers have begun to focus on developing pro-active approaches that attempt to discern whether someone is currently a threat.<\/span><\/p>\n

Techniques that rely on trained experts unfortunately are hard to scale up. But one promising avenue is linguistic analyses that show a statistically significant higher deception detection accuracy than\u00a0the classic and disappointing 54% success<\/a>\u00a0found in the scientific research when people rely on audiovisual cues (i.e. body language).<\/span><\/p>\n

Because NLP has been shown capable of analyzing verbal content, it seems only natural to combine the success of people in linguistic analyses with NLP to see if a hybrid approach is beneficial. For example, in social media content moderation machine learning algorithms flag certain content as problematic and then rely on a person to formally judge it.<\/span><\/p>\n

In \u201cHow humans impair automated deception detection performance\u201d people were asked to assess for deceptiveness and truthfulness the statements made by other people about their most significant \u201cnon work-related\u201d activity for the forthcoming week. Those providing the statements to be examined were asked to either tell the truth or to deceive. Researchers knew the actual biographical details of these statements so that ground-truth was established. In total, 2,027 people provided a statement for evaluation.<\/span><\/p>\n

Furthermore, the participants were asked to evaluate on a scale of 1 to 10 how motivated they were to succeed in convincing others of their deception. This addressed one of the criticisms of some deceptions science research. Namely, that deceivers in a laboratory setting may not be as motivated as those who seek to deceive in real life.<\/span><\/p>\n

The deception scientists in this paper decided to test several different hypotheses in evaluation of the statements they collected:<\/span><\/p>\n

1. Have both people and a machine learning linguistic algorithm, classify\u00a0x<\/em>\u00a0as being either deceptive or truthful.<\/span><\/p>\n

2. Examine the success of the above under four conditions:<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0a. Pure machine learning NLP-based algorithm.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0b. Pure human judgement.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0c. Hybrid Condition 1, people are fully allowed to overrule the assessment of the algorithm.<\/span><\/p>\n

\u00a0\u00a0\u00a0\u00a0\u00a0d. Hybrid Condition 2, people are constrained in how much they may override the assessment of the algorithm.<\/span><\/p>\n

\n

<\/strong><\/em><\/span><\/h4>\n

How humans impair automated deception detection performance<\/strong><\/em>: Major Findings<\/strong><\/span><\/h3>\n<\/div>\n
    \n
  1. Success rates for the above four conditions were:<\/span><\/li>\n<\/ol>\n

    \u00a0\u00a0\u00a0\u00a0\u00a0a. 69% = Pure machine learning NLP-based algorithm.<\/span><\/p>\n

    \u00a0\u00a0\u00a0\u00a0\u00a0b. 50% = Pure human judgement.<\/span><\/p>\n

    \u00a0\u00a0\u00a0\u00a0\u00a0c. 51% = Hybrid Condition 1, people may overrule the algorithm\u2019s judgment.<\/span><\/p>\n

    \u00a0\u00a0\u00a0\u00a0\u00a0d. 67% = Hybrid Condition 2, people may tweak the algorithm\u2019s judgment.<\/span><\/p>\n

    2. Accuracy at detecting deception vs. truth:<\/span><\/p>\n

    \u00a0\u00a0\u00a0\u00a0\u00a0a. 76% deception & 60% truth = Pure machine learning NLP-based algorithm.<\/span><\/p>\n

    \u00a0\u00a0\u00a0\u00a0\u00a0b. 24% deception & 76% truth = Pure human judgment.<\/span><\/p>\n

    \u00a0\u00a0\u00a0\u00a0\u00a0c. 25% deception & 76% truth = Hybrid Condition 1, people may overrule the algorithm\u2019s judgment.<\/span><\/p>\n

    \u00a0\u00a0\u00a0\u00a0\u00a0d. 60% deception & 74% truth = Hybrid Condition 2, people may tweak the algorithm\u2019s judgment.<\/span><\/p>\n

    3. People in the hybrid conditions tended to adjust the judgments of the algorithm more toward truthful. In other words, similar to other work done in deception science, this indicates that people have a truth bias, rather than a deception bias. That is, when evaluating veracity they tend to judge statements as being more truthful than they should.<\/span><\/p>\n

    4. Regarding the motivation of those in the deceive condition, they scored 8.45 on the 10 point scale. In other words, they seem motivated.<\/span><\/p>\n

    \n

    <\/strong><\/em><\/span><\/h4>\n

    Quotes of Note<\/strong><\/em><\/span><\/h3>\n<\/div>\n