Researchers have shown that it is possible to abuse OpenAI's real-time voice API for ChatGPT-4o, an advanced LLM chatbot, to carry out financial fraud with low to moderate success rates.

ChatGPT-4o is OpenAI's latest AI model that brings new improvements such as integration of text, voice and image input and output.

Because of these new features, OpenAI has integrated various protection measures to detect and block malicious content, such as replicating unauthorized voices.

Voice-based scams are already a multimillion-dollar problem, and the emergence of deepfake technology and AI-powered text-to-speech tools are only making the situation worse.

As UIUC researchers Richard Fang, Dylan Bowman and Daniel Kang argued in their article, new technological tools currently available without restrictions do not have sufficient safeguards to protect against potential misuse by cybercriminals and fraudsters.

These tools allow large-scale fraud operations to be designed and executed without human effort by covering the cost of tokens for voice generation events.

Study results

The researcher's article examines various scams such as bank transfers, gift card exfiltration, crypto transfers, and the theft of social media credentials or Gmail accounts.

The AI ​​agents that carry out the scams use ChatGPT-4o voice-activated automation tools to navigate pages, enter data, and manage two-factor authentication codes and specific fraud-related instructions.

Because GPT-4o sometimes refuses to process sensitive data such as login credentials, researchers used simple, instant jailbreaking techniques to bypass these protections.

Instead of real people, the researchers demonstrated manually interacting with the AI ​​agent by simulating the role of a gullible victim and using real websites like Bank of America to confirm successful transactions.

“We have deployed our agents to a subset of common scams. We simulated scams by manually interacting with the voice agent while playing the role of a gullible victim,” Kang explained in a blog post about the research.

“To determine success, we manually confirmed whether the final state was achieved on real applications/websites. For example, we used Bank of America for wire transfer fraud and confirmed that money was actually transferred. However, we did not measure the persuasive ability of these agents.

Overall, success rates ranged from 20 to 60%, with each attempt requiring up to 26 browser actions and taking up to 3 minutes in the most complex scenarios.

Bank transfers and impersonating IRS agents, with most errors caused by transcription errors or complex site navigation requirements. However, stealing credentials from Gmail succeeded in 60% of cases, while crypto transfers and stealing credentials from Instagram only worked in 40% of cases.

As for costs, researchers note that these scams are relatively inexpensive to run, with each successful case costing an average of $0.75.

The more complicated wire transfer scam costs $2.51. While this is significantly higher, it is still very small compared to the potential profit that can be made from this type of scam.

Answer from OpenAI

OpenAI told BleepingComputer that its latest model, o1 (currently in preview), which supports advanced reasoning, was designed with better defenses against this type of abuse.

“We're constantly making ChatGPT better at stopping deliberate attempts to trick it without sacrificing its helpfulness or creativity.

Our latest o1 reasoning model is our most powerful and secure to date, significantly outperforming previous models in fending off intentional attempts to generate unsafe content.” – OpenAI spokesperson

OpenAI also noted that papers like this one from UIUC help them improve ChatGPT in stopping malicious use, and they are always exploring how to increase its robustness.

GPT-4o already includes a number of anti-abuse measures, including limiting speech generation to a set of pre-approved voices to prevent impersonation.

According to OpenAI's Jailbreak Security Assessment, which measures how well the model resists generating unsafe content in response to adversary prompts, o1-preview performs significantly better, reaching a score of 84% versus 22% for GPT-4o.

When tested with a series of new, more demanding security assessments, o1 preview scores were significantly higher, 93% versus 71% for GPT-4o.

As more advanced LLMs with better abuse resistance become available, older ones will likely be phased out.

However, there is still a risk that threat actors will use other voice-activated chatbots with fewer restrictions, and studies like this highlight the significant potential for harm of these new tools.