{"id":14161,"date":"2021-03-02T12:04:51","date_gmt":"2021-03-02T17:04:51","guid":{"rendered":"https:\/\/jasonapollovoss.com\/web\/?p=14161"},"modified":"2025-09-02T12:41:23","modified_gmt":"2025-09-02T16:41:23","slug":"d-a-t-a-is-double-blind-tested","status":"publish","type":"post","link":"https:\/\/jasonapollovoss.com\/web\/2021\/03\/02\/d-a-t-a-is-double-blind-tested\/","title":{"rendered":"D.A.T.A. is Double Blind Tested"},"content":{"rendered":"

<\/span><\/p>\n

By Jason Apollo Voss, CFA<\/em><\/span><\/p>\n

At Deception And Truth Analysis (D.A.T.A.), Inc. we validate our work using multiple and varied methods to ensure the efficacy of our algorithms. We have a preferred method for double-blind testing our work using a gold standard database for testing such algorithms. The database is a collection of close to a thousand proven false statements from the George W. Bush administration in the lead up to the Iraq War from 11 September 2001 to 11 September 2003. The database is considered by deception scientists to be the gold standard because it also contains an equal number believed to be true statements from the same speakers, and spoken at the exact same time, frequently during press conferences, television interviews, or in presentations.<\/span><\/p>\n

Because the database contains both deceptive and truthful statements, from the same people, and at the same time it is a brilliant way of testing the efficacy of any deception detection algorithm and its ability to discriminate between deception and truth in language. It also is a gold standard database because these are real world deceptions with extraordinarily high stakes. If the George W. Bush officials were able to convince people of their narratives there would be a war with Iraq in 2003. If not, there would be no war. These contrast with the more typical artificial testing-environments manufactured in university research labs with college students.<\/span><\/p>\n

In any case, we are thrilled to report the following total accuracy rates for D.A.T.A. by the total word count of a given statement – if you are unfamiliar with thinking of things in terms of word count, most book-sized pages are ~250 words.<\/span><\/p>\n

\n
<\/span><\/div>\n<\/figure>\n

To put these results in perspective, the best previously reported success rate in the scientific literature on text-based deception detection is 72.74%. Importantly, that research did not disclose the word counts of texts under consideration so we cannot directly compare our results. It is important to point out that not only is our algorithm more performant than any other of which we are aware, we have also designed our algorithm to work in almost any setting rather than being optimized for a single task, such as evaluating potlitical statements. D.A.T.A. is grounded in the over one hundred years of deception science into the behaviors of deceivers and truth-tellers. We then use NLP to provide an assist in evaluating these behaviors present in documents.<\/span><\/p>\n

Above we have highlighted the 800 word count level because we believe that if a client assesses a given document of at least that word count that we can render a reliable result of 88.41% overall accuracy. At the >800 word count level the success rate of deception detection is 88.7%, with a false negative of 11.3% (i.e.Type I error, we say a document is deceptive, when, in fact it is truthful). By contrast the truth detection rate at >800 word count is 85.7%, with a false positive rate of 14.3% (i.e. Type II error, we say a document is truthful, when in fact it is deceptive).<\/span><\/p>\n

For those of you more statistically-minded, our p-value at the 800 word count level and above is 1.41648534942489 E-11 and Cohen’s\u00a0d<\/em>\u00a0is 4.23. Our overall double-blind tested accuracy for all word counts is 63.7%, with a Type I error of 34.7% and a Type II error of 35.9%. Our overall p-value is 9.16228417140376 E-25 and Cohen’s\u00a0d<\/em>\u00a0is 2.75.<\/span><\/p>\n

Because of this validation, as well as the other methods we have used to validate D.A.T.A., we are confident that our algorithms accurately discriminate between truthful and deceptive statements and serve as an aid to our Clients in making key, scaled, due-diligence decisions where they are reliant on the representations made by people.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"

By Jason Apollo Voss, CFA At Deception And Truth Analysis (D.A.T.A.), Inc. we validate our work using multiple and varied methods to ensure the efficacy of our algorithms. We have a preferred method for double-blind testing our work using a gold standard database for testing such algorithms. The database is a collection of close to […]<\/p>\n","protected":false},"author":2,"featured_media":14162,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_et_pb_use_builder":"","_et_pb_old_content":"","_et_gb_content_width":"","footnotes":""},"categories":[3],"tags":[442,441],"class_list":["post-14161","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-the-blog","tag-data","tag-validation"],"_links":{"self":[{"href":"https:\/\/jasonapollovoss.com\/web\/wp-json\/wp\/v2\/posts\/14161","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/jasonapollovoss.com\/web\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/jasonapollovoss.com\/web\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/jasonapollovoss.com\/web\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/jasonapollovoss.com\/web\/wp-json\/wp\/v2\/comments?post=14161"}],"version-history":[{"count":0,"href":"https:\/\/jasonapollovoss.com\/web\/wp-json\/wp\/v2\/posts\/14161\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/jasonapollovoss.com\/web\/wp-json\/wp\/v2\/media\/14162"}],"wp:attachment":[{"href":"https:\/\/jasonapollovoss.com\/web\/wp-json\/wp\/v2\/media?parent=14161"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/jasonapollovoss.com\/web\/wp-json\/wp\/v2\/categories?post=14161"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/jasonapollovoss.com\/web\/wp-json\/wp\/v2\/tags?post=14161"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}