Major retailers say it won’t be long before sophisticated AI “assistants” plan your meals, organise your parties and do your shopping.
But companies, many that are already struggling with their more primitive AI chatbots, will have to balance making the newer, “agentic” bots relatable without them going rogue.
AI chatbots were in the news recently when Woolworths reined in its virtual shopping assistant, Olive, after the company’s attempt to have the robot relate to customers on a human level backfired.
Customers reported feeling annoyed rather than soothed when Olive told them about its “relatives” over the phone.
Sign up: AU Breaking News email
As one complained on Reddit: “I’m already pissed that I have to call and now I’ve got some robot babbling to me on the phone? Wtf Woolies?”
While Woolworths has said it will dial down Olive’s quirky personality, the incident – and further testing by Guardian Australia of a range of retailers’ chatbots – shows the technology still has teething problems.
The supermarket’s snafu follows a growing list of AI customer service mishaps, including Bunnings’ chatbot offering illegal electrical advice and Air Canada’s virtual assistant incorrectly promising a bereavement fare refund.
ASX-listed companies Woolworths, Coles and Wesfarmers (owner of Bunnings, Kmart, Officeworks and Priceline) are among the businesses that have announced plans for agentic shopping assistants.
There’s plenty of hype. In a 2024 report, business consultancy Accenture gushed that “consumers are ready” for generative AI-powered shopping assistants, while encouraging companies to make decisions with a “delightfully human” mindset.
Even if consumers are ready, is the technology?
A customer service transformation
Online chatbots meant to help customers have been around for a while, but the tools are becoming more sophisticated.
Primitive versions were built using “rules-based” AI, says Uri Gal, a professor of business information systems at the University of Sydney.
This type of chatbot follows a “decision tree” in order to offer immediate answers to basic questions, Gal says.
For example, if a customer asks, “How do I return my order?”, the bot will typically direct them to the retailer’s returns page or cite the policy.
When “given a certain prompt, it will always give you the same response”, Gal says.
The newer AI-powered retail bots can “learn” new information based on what they are told and generate different answers.
They are often built using one of the big tech companies’ large language models (LLMs), such as ChatGPT.
The next frontier is agentic AI shopping assistants designed to mimic human behaviour.
Gal says these agents “act on their own, as it were, to try and achieve objectives without specific prompts along the way”, such as purchasing airline tickets or groceries.
Gal says agentic AI operates with more ambiguity, which comes with an added level of risk, including privacy concerns if the bots have greater access to customers’ data so they can act more autonomously.
“Given the novelty of these systems, and as we’ve seen just now in the case of Woolies, there’s obvious governance issues that haven’t been really worked out by these organisations,” he says.
“It’s kind of safe to anticipate that different things will happen, which might be risky or interpreted as an agent going rogue.”
Woolworths has partnered with Google to use its LLM, Gemini, to transform Olive into a “shopping companion” that can perform more complex tasks, such as helping plan meals and parties, and automatically add items to customers’ baskets.
The supermarket has said Olive’s more advanced capabilities will roll out at a later date, but its partnership with Google has already allowed the bot to take phone calls – evidently with mixed results.
Woolworths was contacted for comment.
When things go wrong
In Woolworths’ case, first reported by the Sydney Morning Herald, the supermarket said Olive wasn’t glitching or going off-piste on its own.
Instead, a staff member had programmed the bot to talk about its “mother” in response to a customer providing their birthdate, in an effort to give it a personality, the supermarket said.
“As a result of customer feedback, we recently removed this particular scripting,” a Woolworths spokesperson said.
Generally speaking, Prof Jeannie Paterson, the co-director of the University of Melbourne’s Centre for AI and Digital Ethics, says AI assistants get things wrong when they misunderstand a prompt.
“Chatbots are only as good as their ability to decode or understand – and I hate the word understand, because they’re not alive – what it is this human is getting at,” Paterson says.
Last year, Bunnings was criticised after its AI chatbot told a Queensland customer how to rewire an extension cord, despite it being illegal for them to do so without an electrician’s licence.
In 2022, Air Canada’s chatbot incorrectly told a passenger they could buy tickets at full price and later apply for a bereavement fare refund, when no such policy existed.
When Air Canada refused to honour the chatbot’s advice, the passenger sued the airline and won, despite the airline trying to claim in its defence that the chatbot was a “separate legal entity”.
Paterson says companies are “clearly responsible” for their chatbots.
She says businesses try to strike a delicate balance between having a responsive, adaptable AI assistant and the risk of the bot going rogue, or providing incorrect advice that could cost them money.
“One person’s AI agent buying too many eggs or too much salmon isn’t a problem,” she says. “But what if every chatbot across the network does that? You can see that there’ll be lots and lots of money lost before they even address it.”
To mitigate this risk, she says businesses generally put “really strict guardrails” on their bots, which means they are less flexible and worse at interrogating the intention behind a customer’s prompt.
Guardian Australia tested a range of retail bots, which delivered marginal results, suggesting the technology is still in its infancy.
In one example, when Uniqlo’s “virtual shopping assistant” was told: “I am looking for a woollen jumper”, it replied: “Sorry, we could not recognise you.” After entering “find a product” and then “woollen jumper”, it came back with a range of men’s button-down office shirts. Uniqlo was contacted for comment.
Even Olive wasn’t on the money. Asked via the Woolworths’ chat function: “How much is a 500g bag of pasta?”, the cute anthropomorphic olive replied: “I’m very sorry to hear you were missing items from your order.”
