I’m paraphrasing this essay:

Potemkin AI: Many instances of “artificial intelligence” are artificial displays of its power and potential

Algorithms might also be black-boxed through the force of law by the tech companies who claim them as trade secrets. In The Black Box Society, Frank Pasquale details how many of the algorithms that govern information and finance —the circulation of data and dollars — are shrouded in opacity. Algorithms are often described as a type of recipes. Just as Coca Cola keeps their formula a tightly guarded secret, so too do tech companies fiercely protect their “secret sauce.” Again, it’s one thing to enjoy a beverage we can’t reverse-engineer, but quite another to take on faith proprietary software that makes sentencing decisions in criminal cases.

Potemkin AI is related to black boxing, but it pushes obfuscation into deception. The Mechanical Turk, like many of the much-discussed AI systems today, was not just a black box that hides its inner workings from prying eyes. After all, Kempelen literally opened his automaton’s cabinet and purported to explain how what looked to be a complex machine worked. Except that he was lying. Similarly, marketing about AI systems deploy technical buzzwords work as though they were a magician’s incantations: Smart! Intelligent! Automated! Cognitive computing! Deep learning! Abracadabra! Alakazam!

Weaving the right spell can endow an AI system with powers of objectivity, neutrality, authority, efficiency, and other desirable attributes and outcomes. Like any good trick, it matters less if the system actually works that way than if people believe it does and act accordingly.