In today's e-commerce ecosystem, chatbots have become the first line of customer service. However, many audits stay on the surface: they catch errors, but rarely check whether the proposed fixes actually work. In this article, we look at a real-world case where an e-commerce support bot went from a rating of 86 to 91 after applying specific adjustments, and explore how this test-and-re-evaluate approach can transform a business's user experience and operational efficiency.
The initial audit was conducted using a testing system with four customer profiles—curious, frustrated, confused, and technical—that interacted with the bot in typical scenarios: order tracking, returns, refunds, sizes, and general questions. The result was an 86 out of 100, a B grade that indicated a solid performance but with correctable gaps. What was interesting was not the bugs themselves, but the subsequent process: instead of just listing bugs, the suggested fixes were applied and the exact same battery of tests was run again. The second result was a 91, grade A, with tangible improvements in categories such as accuracy, adherence to the profile and robustness.
What exactly was corrected? On the first run, the bot would ask for an order number from a frustrated user whose package hadn't arrived for two weeks, only to later admit that they didn't have real-time access to the data. That generated a false expectation and increased the customer's anxiety. The solution was to immediately redirect to the tracking link without requesting information that could not be processed. Another recurring glitch was that the bot never mentioned item condition requirements in returns—without saying they should be unworn, unwashed, and tagged—leading to warehouse rejections and disputes. Ambiguity was also detected at the exact time of the refund: the bot said "after receiving the item" without clarifying whether it was after the carrier's scan or the warehouse's confirmation, a crucial difference for the customer tracking their money. Finally, when a user mentioned that they had purchased "about a month ago", the bot would initiate the return process without warning that they might be outside the 30-day period.
The change applied was a direct update to the system prompt, instructing the bot not to ask for order numbers that it could not query, to always mention the condition requirements of the items, to specify that the refund begins after confirmation from the warehouse, and to proactively warn about the eligibility timeframe. Without modifying anything else, the re-test showed that each of these instructions was consistently complied with in all profiles. The most notable improvement was in the robustness category, which went from 78 to 88, precisely because the original flaws were related to the handling of ambiguities, missing data, and edge cases.
This case illustrates a fundamental lesson for any company that implements AI agents in its customer service: it is not enough to diagnose, you have to validate that the solutions work in real conditions. At Q2BSTUDIO, we understand that AI for business must be continuously calibrated with actionable scenarios. Our team develops custom applications that integrate chatbots, virtual assistants and automation systems, always incorporating audit and re-test cycles to guarantee measurable results.
In addition, the scalability of these solutions depends on a solid infrastructure. We work with AWS and Azure cloud services to deploy AI models securely and efficiently, and we complement the experience with data analytics using business intelligence services and Power BI, allowing companies to visualize the real impact of their customer service improvements.
Cybersecurity also plays a critical role. When auditing a bot that handles order and payment data, it's vital to ensure that information flows don't expose vulnerabilities. We offer cybersecurity and pentesting to protect every layer of the system, from user interaction to the database. In the case analyzed, part of the improvement in robustness came from avoiding questions that forced the bot to admit lack of access, a symptom of an insecure or poorly integrated design.
The re-test approach is not a luxury, but a strategic necessity. Many companies invest in AI for businesses and launch chatbots without validating that the fixes proposed by audit tools actually solve problems. As this case demonstrates, a set of specific changes to the system prompt can generate a measurable improvement of 5 points in the overall rating, and up to 10 points in robustness. But the real value is in the methodology: knowing that each fix has been tested against the same scenarios that revealed the flaw.
For companies looking to optimize their customer service processes, the recommendation is clear: implement a continuous cycle of auditing, correction and re-evaluation. Whether it's a returns bot, a sales assistant, or a helpdesk system, combining process automation with well-calibrated AI agents reduces costs, improves satisfaction, and avoids disputes. At Q2BSTUDIO we help companies design and integrate these solutions, from cloud architecture to business metrics analysis, ensuring that each improvement is validated with real data.
In short, the step from 86 to 91 is not just a number; It is the demonstration that a useful audit does not end in the diagnosis, but in the verification that the solutions work. Companies that adopt this approach – supported by developers who specialize in custom software and AI strategies – will be better prepared to deliver consistent, reliable, and scalable customer experiences.
