A Gartner report has made a highly troubling prediction that could drastically alter the role of generative artificial intelligence in the customer service sector.
Listed as one of its Strategic Planning Assumptions, Gartner stated that “by 2027, a company’s generative AI chatbot will directly lead to the death of a customer from bad information it provides.”
In expounding on this prediction, Brad Fager – Senior Director and Chief of Research at Gartner – explains that the potential threat will come from “hallucinations”, a term used by Open AI to describe content created by GenAI that is “nonsensical or untruthful in relation to certain sources.”
Ironically, it is the more human and realistic capacity of GenAI that has been integral to its success, which could enhance the believability of these hallucinations.
As customers become more trusting of authentic-feeling chatbots, they will become over-reliant on the technology and not be able to discern irrelevant and potentially dangerous advice/suggestions.
The idea of customers being placed in life-threatening situations by technology will undoubtedly remind some readers of the 2014 Apple scandal, which saw a fake ad claiming that Apple products could be charged in a microwave. Worryingly however, the threat from GenAI is a much larger concern.
Whereas the fake Apple ad was perpetrated by hackers as a prank, Gartner has warned that future instances of placing customers in danger “could come from any generative AI tool.”
Gartner has warned that future instances of placing customers in danger "could come from any generative AI tool."
Rather than intentionally looking to trick people, the concern with GenAI is that it could provide seemingly good advice that is applied in an improper way – whether it be neglecting certain food allergies or dangerous product repair suggestions.
While one would assume that safeguarding against examples such as those listed above would be at the top of any company’s list before implementing a new GenAI tool, the proliferation of generative AI at such a remarkable speed has completely outpaced risk assessors.
Indeed, in Gartner’s 2Q23 Emerging Risks Report, a survey revealed that “Only 1% of risk executives said they were thoroughly prepared for the risks of AI adoption.”
In spite of these risks and concerns, it is clear that the GenAI gravy train won’t be slowing down any time soon. There is no point in closing the stable door after the horse has bolted.