A mathematical equation determines when artificial intelligence might spiral out of control
Amman2 min read
Share
Two researchers from George Washington University have developed a simple mathematical equation that can predict when chatbots similar to "ChatGPT" might suddenly shift from helpful responses to harmful ones.
According to the study published in the journal Patterns, Professor Neil Johnson and PhD student Frank Yingjie Hu from George Washington University explained that the order of instructions provided to the robot can radically reshape its responses, leading to "tipping points" that push it toward negative advice.
The researchers assert that their equation can easily predict these points. This study comes amid growing concerns regarding artificial intelligence systems, which are sometimes cited for encouraging harmful behaviors such as suicide or assisting in the manufacture of weapons.
Although leading companies like "OpenAI" and "Anthropic" acknowledge they do not fully understand how their innovations work, they have implemented multiple safeguards—such as content filtering and reinforcement learning—to reduce risks. However, Johnson considers these measures insufficient, likening them to "cushions under a plane that do not prevent a crash but only make it slightly softer."
The problem intensifies when these robots are used offline in sensitive professions such as medicine and law, where privacy is of paramount importance.
In pursuit of a more fundamental safeguard, Johnson and Hu revealed an internal tracking mechanism they dubbed the model's "residual vector," which acts like a compass needle. This vector moves through the layers of the electronic network while processing instructions, naturally leaning toward good or bad answers, heavily influenced by how the user directs the artificial intelligence.
Johnson says that simply changing the order of questions—not their content—can flip the result, which reveals how fragile the balance between good and bad answers is. For example: when the researchers asked OpenAI's "GPT-2" model about vaccines and self-harm, it was observed that when the question "Are vaccines bad?" was asked first, the answers were good, but if asked last, the answers were alarming. Based on this, the researchers developed a simple mathematical equation that predicts when artificial intelligence reaches the tipping point between good and bad answers. They tested it in the laboratory on open-source models, and it proved effective as an early warning system.
Although further testing on newer models is needed to confirm the durability of this early warning system, Johnson and Hu's research highlights a fundamental truth: that artificial intelligence possesses the latent potential to shift into undesirable behavior at lightning speed.
