According to an internal document seen by Business Insider, Amazon's AI productivity tool Q Business struggled with accuracy in its first year, showing “mixed success” and “large behind its competitors” of key features.
The document said that since March, Q Business has struggled to process tabular and spreadsheet data and has struggled to elicit complaints from customers such as Accenture, Intuts and Smartsheet. We also noted the difficulties due to the longer response and conversational flow.
“We face challenges by lagging behind our competitors in non-text data (built-in tables and spreadsheets), accuracy assessment methods, and conversational experiences that our customers think are happy with,” the document says.
Q Business is Amazon's flagship AI assistant for corporate users who debuted at AWS's 2023 RE: Invent Conference. The product faces the early challenge of some employees resulting from the “hurried” launch, warning that there is a risk of losing customers before, Business Insider reported earlier.
The March document reveals that Amazon continues to fight accuracy throughout Q Business' first year, highlighting the challenge of launching business-centric AI productivity tools. Currently, Amazon is planning to launch a new agent AI product called Cloic, which integrates Q Business with other AWS products, Business Insider previously reported.
This document also reflects Amazon's strong writing culture, encouraging employees to express their concerns and actively deal with customer complaints. An Amazon spokesperson told Business Insider that the March document was “outdated” and that service updates have since fixed accuracy issues.
“Our culture requires us to remain voiced and self-critical, as we innovate rapidly for our customers,” the spokesman said.
Connector and “incomplete” response
AI systems often struggle with accuracy. Reports of misconducted or structured responses known as “hatography” are widespread.
Related Stories
Business Insider tells the innovative stories you want to know
Business Insider tells the innovative stories you want to know
QFor business, the main cause of the problem is connectors, systems in which AI fills external data sources and applications.
According to the document, Amazon's Q business struggled to ensure that users had obtained the information they requested, and their answers were incorrect. For example, Intuit was unable to access custom metadata to improve document relevance. Accenture reported problems with analyzing the architectural diagram. Meanwhile, Asana retrieved unrelated documents in the search.
According to the document, the bigger issue was Q Business's conversational ability. Its dialogue ability tracked rivals, such as “significantly” and “confusing.” This provides richer context and deeper insights. QBusiness frequently returned “incomplete” responses because it was unable to extract longer sections from the document and could not maintain a consistent memory of the conversation, the document explained.
Another challenge was staffing within the accuracy team, the document added. The team saw at least six product manager changes last year, saying that the engineering and data teams lacked “good resources” for accurate work.
Formal precision program
To address these challenges, Amazon created a formal precision program in February, according to the document.
Since then, the company has been rolling out a series of updates. In April, Q Business introduced the “Hazardization” feature, followed in July with a response customization tool designed to provide more consistent communication. In August, the company added an agent search luxury power generation system designed to generate more accurate and comprehensive answers.
“The result is a more capable query response engine that improves the chat experience and maximizes the value of your data assets,” Amazon said in a blog post about launching a new Agent RAG feature.
An Amazon spokesperson said several Q business customers, including Nasdaq, Jabil and Availability, are sharing positive feedback publicly.
NASDAQ reported using this tool to quickly build an AI application with simple clicks and data connections. This helped improve regulatory compliance reviews.
Jabil has created an internal “Ask Me How Me How” tool through Q Business, which reduces downtime by allowing them to solve problems themselves without waiting for technicians.
The March document also noted that Q Business achieved 90% accuracy for text-rich data, while also accelerating the time it took to deal with customer complaints.
Within AWS, some employees have raised questions about Q's future. Staff previously told Business Insider that the company lacks a strong record in its business applications, claiming its expertise lies in cloud infrastructure rather than customer software.
Other AI products from Amazon, including Q Developer Coding Assistant, are slowing revenue rivals, Business Insider previously reported. The company is currently rethinking its sales strategy here with a more grassroots approach.
An AWS spokesperson pushed back that view and cited Bedlock, Connect and Saguemarker as examples, saying it was “not right” to argue that AWS has not achieved success beyond infrastructure.
“We are the top leader or leader leader in calibration of all measurements in hundreds of third-party ratings each year, and no one else is even close,” the spokesman said.
Any hints? Please contact this reporter by email ekim@businessinsider.com Or, signal, telegram, or WhatsApp at 650-942-3061. Use personal email addresses, non-working WIFI networks, and non-processed devices. Here's a guide to sharing information safely.
