AI promises to reimagine the contact center by automating contacts, elevating employees, and redefining experiences.
However, AI is not just delivering new, game-changing capabilities to service teams; it's also bringing new tools to attackers.
Recognizing this, contact center leaders must understand emerging threats to their operations and customers.
As such, CX Today reached out to Santosh Kumar, Chief Security Architect at Cisco, to identify six new risks of AI and how to combat them.
1. AI Voice Phishing
In February, a startup named "Zyphra" launched two open text-to-speech (TTS) models, each capable of cloning someone's voice with as little as five seconds of sample audio.
An impressive achievement? Absolutely. But one with several risks to many businesses.
After all, with such technology, a fraudster may conduct a voice phishing attack that can convincingly bypass voice biometric systems.
For instance, an attacker could call a bank, pass the voice recording as authentication, and gain full access to the account.
That may seem far-fetched, but—in November—a BBC journalist successfully used voice cloning technology to bypass voice ID systems at two prominent UK banks.
The threat is significant. Indeed, OpenAI stalled the release of a similar solution last year, warning businesses to “phase out voice-based authentication”.
Commenting on this threat, Kumar noted: “The growth of AI-driven voice phishing has increased by 3,000 percent compared to two years ago.
“To mitigate this, it's crucial to implement anti-spoofing mechanisms, multi-factor authentication, and liveness tests to verify the caller's presence.”
Companies that haven’t already implemented similar voice biometric protections are especially vulnerable to this AI threat.
2. Privacy Risks
The growing use of machine learning (ML) models in contact centers introduces new challenges. These go beyond the scope of traditional practices--like encryption, access controls, and GDPR compliance--which, of course, remain essential.
Yet, businesses must consider new practices to protect against new breaches of these models.
For instance, there are “membership inference attacks”, where a fraudster attacks an ML model by inputting specific queries to determine if certain individuals’ data were used in their training.
In doing so, the attacker may access that individual’s personal information.
Additionally, they may gain insight into how the model was trained. That could allow them to tamper with it or create a fraudulent duplicate – as scammers are doing more and more.
To mitigate such AI threats, Kumar advises against leveraging machine learning models trained on small datasets and ensuring the model has gone through adversarial testing.
“Every model in our pipeline undergoes adversarial testing before deployment,” said Kumar.
“We also explore differential privacy techniques to ensure prediction vectors remain ambiguous, preventing attackers from extracting precise information.”
Remember, ML models often memorize sensitive data, so always treat them cautiously.
3. Chatbot Attacks
Chatbots offer a common entry point for attacks, especially those powered by machine learning. After all, they can be targeted by adversarial attacks like those highlighted above.
Yet, as businesses power bots with large language models (LLMs), there's now a risk of "prompt injection" attacks. These are either direct – aiming to trigger specific responses – or indirect – striving to change the virtual agent's behavior.
Via both methods, users can trick the bot into performing prohibited tasks.
These attack methods received widespread publicity after security researcher Johann Rehberger used similar techniques to tamper with Google Gemini's long-term memory.
However, there are other chatbot attacks to guard against. For instance, a fraudster could manipulate the bot into adopting a persona. Alternatively, they may exploit AI’s limited context window to overload it with irrelevant data, hampering its performance.
