User experiences and expectations are changing with the growth in chatbots and there is an element of self-empowerment to solving problems as well as not wanting to interact with someone in a call centre. A study by Salesforce found that 72% of Millennials did not believe a phone call was the best way to resolve their issue.
How do we define ‘helping’?
But before we answer the question, ‘Is your chatbot helping your customers?’, we need to set a measure of desired outcome. Are we after containment, or overall satisfaction? Clearly we have to put customer satisfaction at the top and resolving queries first time up there too. Containment will naturally follow although there should be caution to complex or multiple queries where it is difficult for the customer to self-diagnose, and experience frustration online.
We also have to recognise the limits of even the perfect chatbot: If you have a complaint, how can chatbots help? Can they exercise judgement to distinguish customers that are trying it on? Can they really troubleshoot rare, complex and personal problems that banks typically deal with?
One of the reasons why chatbot learning hasn’t progressed as fast as the media would like is the intensity of the process and the machine learning modelling process: New York Times estimated that up to 80% of a data scientist’s time is spent “data wrangling”. CrowdFlower estimates “data preparation” at 80%.
Further, assumptions and errors are an inevitable part of the process where human judgement and skill is required. Someone needs to build a training set for the chatbot and that typically involves a human labelling the chat interactions into topics, creating a topology that can drive the correct response. This is laborious in a linear fashion.
Banks also need to assess whether their systems accommodate automation technology. Does the chatbot have the data it needs to make accurate responses? Does it have the right humans to train it? Is the organisation able to act on the insights generated from chatbot activity? Will the technology integrate with other systems such as CRM?
What do others think about their chatbots?
Our own research showed that 59% of businesses who have a chatbot are unsatisfied with its performance.
551 professionals involved in the development or management of chatbots were surveyed by Warwick Analytics.
When discussing the technical challenges respondents faced trying to improve their own chatbots, the most common issues were improving containment rates (90%), reducing errors (83%), and developing the responses for the chatbot (79%).
More significantly, an overwhelming 93% believed that human validation and/or curation was important to maintain and improve the performance of their chatbots.
In addition, 21% of respondents who were yet to deploy a chatbot said it was because the performance of chatbots wasn’t acceptable in their opinion.
Making sure your chatbot is helping
There are a number of different approaches to the problem. Warwick Analytics, a spin-out from The University of Warwick, has developed a proprietary technology that automatically classifies the chatbot’s customer interactions but when its certainty is low, it asks for assistance from a human to classify or ‘label’ the interactions and uses the information to train the models itself. It minimizes the human interaction to maximise the performance of the models.
The models within the chatbot will be accurately tagged and able to retrieve the next best response (or indeed hand off to a human appropriately). The human trainer can be offline, as well as involving the customer in certain circumstances.
Achieving the right level of human-in-the-loop input is key for chatbot owners and managers. Human validation is required for accuracy and improvement but if too much is required then a business may as well have a human service desk. It’s all about finding the right technology that minimises the human intervention required but still increases accuracy.
So, with the right AI and machine learning chatbots can definitely help customers and improve their capability to understand their own uncertainty. In the background, there’s no magic solution but there is a whole bunch of disruptive technology making sure that the chatbot satisfies and doesn’t fail.
By Dan Somers, CEO of Warwick Analytics