Chainalysis' head of investigations doesn't seem to have a great understanding of whether her company's flagship software even works.

Elizabeth Bisbee, head of investigations at Chainalysis Government Solutions, testified she was “unaware” of scientific evidence for the accuracy of Chainalysis’ Reactor software used by law enforcement, an unreleased transcript of a June 23 hearing shows.

The fact that Chainalysis’ blockchain demystification tools have become so widespread is a serious threat to the crypto ecosystem. Although industry insiders have raged against Chainalysis since it was founded, often accusing it of violating people’s financial privacy, there may be a better argument to make against the company and analysis firms like it: it’s within the realm of possibility that these surveillance machines don’t work as well as advertised.

This is a big deal considering Chainalysis' surveillance tools are used widely across the industry for compliance, and have at times led to unjustified account restrictions and – even worse – land unsuspecting individuals on the radar of law enforcement agencies without probable cause.

That’s precisely the argument that renowned lawyer Tor Ekeland is making in his latest defense of an accused early bitcoin adopter, and why he was quizzing a Chainalysis executive on the stand.

Bisbee was testifying in a case between the U.S. government and Roman Sterlingov, the alleged creator of the once popular Bitcoin Fog cryptocurrency mixer used to anonymize bitcoin transactions. Chainalysis’ Reactor software was used to track cryptocurrency payments in Sterlingov’s criminal investigation, and is now being challenged by Sterlingov’s defense.

Sterlingov is represented by Ekeland, who has made a career out of defending hackers and technology providers. Ekeland said Chainalysis’ Reactor is “a black box algorithm” that “relies on junk science.”

In a hearing aimed to establish the admissibility of expert testimony, Bisbee was pressed for details on the accuracy of the Reactor software Chainalysis sells to governments for law enforcement purposes, including what evidence the company has that suggests it works.

"[W]e still live in a democracy in which criminal convictions prerequisite the existence of scientific evidence"

Bisbee said she was unable to provide the court with statistical error rates for Chainalysis’ Reactor software. She further denied being aware of any scientific peer-reviewed papers or “anything published anywhere” attesting to the accuracy of Chainalysis Reactor.

Instead, Chainalysis reportedly judges its software’s accuracy using customer feedback, she said.

Bisbee’s statements are in line with a blog post published by Chainalysis competitor Coinbase, which describes blockchain analytics as “more of an art than a science.” Coinbase offers blockchain analytics services to law enforcement via its Tracer software.

For Chainalysis' part, the company noted in a July 18 court declaration there is no doubt about the Reactor software's findings. Although not "peer reviewed" in an academic sense, the company's clustering heuristic, the algorithm used to find relationships between blockchain addresses, comes to "deterministic" conclusions that can be independently verified and reproduced.

The same statement noted the company is unaware of margins of error rates for the Reactor software, and has not gather and record false positive and negative rates for its software overall.

Unfortunately for Bisbee and her corporate overlords, we still live in a democracy in which criminal convictions prerequisite the existence of scientific evidence. Maybe Bisbee would be better suited pursuing an art history degree.

This article originally posted on CoinDesk.

170 +