AI Helped Lower-Rated Service Agents Gain Ground

A generative AI assistant gave thousands of customer-service workers a starting point—and the workers with the lowest customer ratings before the experiment gained the most.

In a four-week randomized field experiment on Taobao, 5,940 early-tenure after-sales agents handled about 2.56 million customer chats. Half were assigned access to an assistant that proposed an issue diagnosis and a possible solution during the opening stage of each conversation. The suggestions were visible only to the worker, who could send them, edit them or ignore them.

The result was not a simple story of automation. Average service became a little faster and better rated. The largest rating gains appeared among agents in the lowest pretreatment-rating quintile. At the same time, the highest-rated group saw service quality decline. The same tool narrowed one performance gap by helping at the lower end and hurting at the top.

Research references: [1]

Faster service, with a modest average quality gain

Random assignment makes the effect of access the cleanest headline. Agents offered the assistant identified customer issues 8.2% faster and shortened total chat duration by 1.1%—about 3.2 seconds and 5.7 seconds, respectively. Average ratings increased by 0.042 points on a five-point scale, while the share of one- and two-star ratings fell by 0.012. The estimate for customers returning with the same issue within three days was near zero and not statistically significant.

That split matters. A quicker, clearer interaction can improve the customer’s experience without expanding what the worker is authorized to do. Alibaba’s operating procedures stayed the same during the experiment, so the assistant did not create new remedies for unresolved orders.

Research references: [1]

The biggest gains came where support was most useful

The researchers grouped agents into quintiles using their average customer rating before the experiment—a practical proxy, not a complete measure of skill. For agents in the lowest-rated fifth, access raised ratings by 0.812 points and reduced the dissatisfied-rating share by 0.215. Both estimates were statistically significant. Benefits shrank higher in the distribution and became statistically insignificant for the fourth quintile.

For the highest-rated fifth, the direction reversed. Ratings fell by 0.283 points, dissatisfied ratings rose by 0.070 and three-day same-issue returns increased by 0.009. Those access effects were statistically significant.

This is why “AI raised the average” is not enough. The distribution shows who gained, who did not and where a uniform rollout could make an already effective workflow worse.

Research references: [1]

A tool workers could choose how to use

The assistant combined Qwen language models with customer-support systems trained for Taobao’s environment. It produced two pieces of text: what the customer’s issue appeared to be and what the next step might be. Workers retained the final decision.

Access did not mean automatic adoption. In the treated group, an AI suggestion was used in 21.5% of chats: 17.1% were sent without modification and 4.4% were edited. At the agent-day level, average use was 28.7%. Use also differed by baseline group, from 39.3% among the lowest-rated quintile to 18.3% among the highest-rated.

Because workers chose when to use the tool, the authors separately estimated the randomized effect of access and an instrumental-variable estimate of use induced by access. The access results are the firmer basis for deployment decisions. The usage estimates apply to workers whose behavior changed because they received access and depend on additional assumptions.

Research references: [1]

The strongest workers may need a different design

Why did the highest-rated group lose ground?

The paper’s process evidence points toward workflow disruption, but it does not identify a single causal mechanism. These agents often handled multiple chats at once. In instrumental-variable estimates of use induced by access, their overall response time rose 8.6%, and customer returns within ten minutes increased by 0.035. Their later-stage shift-away time also increased. The authors interpret that pattern as consistent with attention being pulled away from a focal conversation that was already running well.

That explanation remains suggestive because chat switching was observed, not randomly assigned. The study did not test alternative interface designs that would isolate whether a quieter, optional or expert-specific workflow would avoid the downside.

The practical principle is still powerful: equal access is not the same as good fit. For a worker building an efficient routine, an editable diagnosis and solution can provide scaffolding. For someone whose rhythm is already strong, the same interface can become another interruption. A human-centered deployment should preserve discretion, evaluate outcomes by worker group and allow expert routines to remain intact when the tool adds friction.

Research references: [1]

Measure the experience and the outcome

Only about 15% of chats received a customer rating. The paper reports that access had essentially no effect on whether a customer submitted a rating, and response rates stayed similar between treatment and control groups. That check reduces one selection concern, but ratings still describe the minority of customers who chose to respond.

The experiment also lasted only four weeks, from January 23 to February 19, 2024, on one Chinese e-commerce platform. Every participating agent had less than a year of tenure, and most had worked four or five months. Pretreatment rating is informative, but it is not a complete measure of expertise. The study does not establish long-run learning or deskilling effects.

The current source is an arXiv preprint, last revised July 31, 2026, not a peer-reviewed publication. Two authors are listed as Alibaba employees. The manuscript does not provide a searchable ethics, worker-consent, funding or conflict-of-interest statement, so those governance details remain unresolved in this review.

Within those boundaries, the experiment offers a better vision than “AI for everyone” or “AI for no one.” The technology helped many workers move faster and brought lower-rated agents closer to their peers. Its failure at the top is not a reason to discard the tool. It is a reason to design around the human.

The future of useful work may depend less on whether an organization adopts AI than on whether it can recognize where people need support, where they need freedom and how to protect both.

Research references: [1] [2]