---
title: "How to use AI to draft customer service replies without letting it send them"
description: "We let a language model draft customer service replies from the team's approved macros and from procedures learned from real agent replies, grounded in the customer's actual order data. A human reviews and sends every reply; the model never sends. The guards around the draft matter more than the prompt, and most of what we learned came from guards that were silently not working."
date: 2026-10-11
lang: en
canonical: https://goingstellar.ro/methods/ai-drafted-customer-service-replies/
author: Going Stellar
publisher: GOING STELLAR S.R.L.
topic: E-commerce / Customer service
license: CC BY 4.0
alternate_ro: https://goingstellar.ro/ro/metode/raspunsuri-customer-service-scrise-de-ai/
---

# How to use AI to draft customer service replies without letting it send them

*We let a language model draft customer service replies from the team's approved macros and from procedures learned from real agent replies, grounded in the customer's actual order data. A human reviews and sends every reply; the model never sends. The guards around the draft matter more than the prompt, and most of what we learned came from guards that were silently not working.*

> E-commerce · Customer service · 2026-10-11 · https://goingstellar.ro/methods/ai-drafted-customer-service-replies/

## Key points

- An AI-drafted customer service reply should be saved as a draft for a human agent to review and send; the sending path should refuse to run without consent given outside the command line.
- An order lookup has three outcomes, not two: not searched, searched and found nothing, and search failed. Treating a failed search as an empty result makes the draft tell real customers they have no order.
- Learn support procedures from the replies of real agents before judging agents or drafting replies; assumed procedures turn correct refusals into false mistakes.
- A guard written as patterns in one language does not fire in another; when drafting moves to a new market, every filter has to be rewritten and measured on real messages from that market.
- The store a support ticket belongs to is the page or mailbox the message was sent to, not the brand label the helpdesk displays.

## What AI-drafted customer service replies are for

An AI-drafted reply is a suggested answer to a support ticket that a language model writes and saves as a **draft** inside the helpdesk, where an agent reads it, edits it and sends it. The model never sends anything to the customer. The draft is built from three inputs: the approved reply macros of the support team, the procedures the team actually follows, and the customer's real data (orders, delivery status, tracking link).

The point of drafting is to remove the typing, not the judgement. An agent who opens a ticket and finds a reply that already names the right order, the right tracking link and the right policy answers faster. An agent who finds a fluent reply that invents a delivery date has to catch the invention, which is harder than writing from scratch.

## Step 1: route the ticket before drafting it

**Find the store from the page the message arrived on.** When one support team serves several stores, the store of a ticket is the social page or mailbox the message was sent to, read from the recipient field. The brand label the helpdesk shows on a ticket can be stale; ours was, and an agent once named the wrong store because of it.

**Triage writes tags and priority, never messages.** Our triage step proposes a store tag, a category and a priority, runs as a dry run by default, and with an explicit flag writes only those internal fields. When we started, 99.7% of conversations had no tags at all, so routing was blind. A legal or consumer-authority threat becomes urgent; returns, product problems and cancellations become high priority.

**Score sentiment for ordering, not for verdicts.** A rule-based score gives intensity 3 to a legal threat, 2 to frustration ("third email", "nobody answers") and 1 to a complaint keyword; a model-based pass also catches calm complaints that contain no keyword.

## Step 2: draft from the approved macros and the learned procedures

**Macros set the format and the voice.** The support team keeps its canonical replies in one document: macros for order tracking, returns and refunds, damaged products, defective or missing items, cancellation, order changes, presale questions and thanks, in each market's language, plus 18 presale answers. The draft picks the macro for the ticket's category and language and fills its variables (customer name, order number, tracking number, store, return link) with real data only.

**Procedures are learned from real tickets, not assumed.** A separate step reads well-resolved tickets per category, keeps only the replies written by human agents (the helpdesk's bot and auto-filled form text are excluded) and documents the de facto procedure: the steps, the canonical wording, and when the team declines. The draft engine injects the section for the ticket's category into the prompt.

**Precedent gives tone, not facts.** For each ticket we retrieve the 3 most similar resolved tickets by meaning and show the model how a human agent answered them. The model is told to take only tone and structure from the precedent, never order numbers, names or dates.

## Step 3: ground every claim, and know when the search failed

A draft may state an order status, a delivery date, a price or a size only if that fact is in the context the draft was given. After generation, a deterministic filter looks for invented lookups ("we checked"), invented statuses, invented deadlines and invented dimensions. A draft that trips the filter is regenerated once with a correction; if the second draft still invents, a safe per-category template replaces it.

**An order lookup has three outcomes, not two:** we did not search, we searched and found nothing, and the search failed. Our lookup used to return an empty list on every failure (missing driver, missing credential, unreachable database), which is exactly what a successful search with no results returns. In a simulated database outage on 162 real tickets, 83 (51.2%) carried the context "searched, does not exist", all false, and the draft would have told those customers they had no order with us. After the fix, 0 do; the same 83 get a block that forbids any statement about whether the order exists.

**A draft confirms an action only after the action ran.** Order changes, cancellations and resends are proposed, run as a dry run, and applied only when an agent approves them. Until then the draft says the request has been taken over, not that the address was changed.

**Answer the last message, not the first.** One draft answered a tracking question that the customer had asked days earlier, while the customer's latest message said the parcel had arrived. The transcript now marks the customer's latest message explicitly.

## Step 4: never auto-send, and make that a guard rather than a convention

The team rule is draft-only. The code path that sends a reply live exists, so since 15 September 2026 it refuses to run unless consent is given outside the command line, through an environment variable or a confirmation file. Without it the process stops with exit code 2 before any network call, so a mistyped flag cannot reach customers.

**Escalations get a holding reply and a human.** Legal threats, a refund promised but not executed, very angry customers and repeat complaints do not get an answer on the substance. The engine saves a short holding draft, raises the priority, adds escalation tags and writes an internal note with the problem, the order, the tracking number and the suggested action.

**Reply means close.** When an agent answers a ticket, the agent closes it, escalations included, because the helpdesk reopens the conversation by itself if the customer writes again.

**Public channels are opt-in and stripped.** Comments under posts and ads get a draft only with an explicit flag. A public draft never contains the customer's name, phone number or order number. In any channel with a mixed audience, such as a group chat, the safe default is silence: nothing about customers, orders or business figures goes there.

## The traps we hit and how we catch them now

**The draft endpoint appends.** Saving a draft adds a new one each time and the helpdesk offers no way to delete drafts through the interface we use. Every re-run on the same tickets stacked another draft that agents had to clean by hand. We now finish the prompt first, write once, tag each batch, and skip tickets that already carry the tag.

**Error text was saved as a reply.** When we ran several workers in parallel, the model provider answered with rate-limit errors (HTTP 429) and the error text itself was saved as drafts. The engine now retries with backoff and refuses to save a draft that is empty or starts with an error marker.

**Triage died silently.** The triage job kept reporting success from mid-June to early July while pulling no new tickets: one script wrote to a database at a path that existed only on a laptop, and the current day was marked done at night while still empty. Both are fixed, and the most recent 2 days are now always re-pulled.

**A guard written in one language is dead in another.** When we extended drafting to foreign markets we translated the prompt but not the guards, which were Romanian patterns. The invented-claim filter caught 1 of 6 identical fabrications across languages, and 20 of 20 foreign legal threats did not escalate. Repairing the guards then raised the false-positive rate on 137 real agent replies from 15.3% to 28.5%. Every multilingual change is now measured on the language that already worked as well.

## Where we were wrong

**We judged agents against procedures we had invented.** Our first quality review flagged agents as wrong for refusing returns of unsealed hygiene products. Refusing those returns is the team's policy. That is why the procedures are now learned from real tickets first, and only deviations from the team's own standard count as mistakes.

**We let a batch run before the drafts were good enough.** An adversarial audit of 75 real drafts found problems in 71% of them, 18 of high severity: invented sizes, prices, phone numbers and statuses, and "we checked" with no lookup behind it. We paused the scheduled batch run. Native-speaker evaluators of the foreign-market drafts, 4 of them, all said the drafts should not be sent. The scheduled run stays paused until a start gate passes on the copy that would actually run, because we once found 3 copies of the engine in 3 different versions.

## Limits and checklist

This method does not make drafts correct; it makes wrong drafts rarer and easier to spot. A model can still misread a ticket, and the guards catch only the kinds of invention someone thought to write a rule for. Our quality audit, which flags slow first responses (over 24 hours by default), long back-and-forth threads, frustration and escalations, found at least one flag on 27% of real tickets in a 30-day window; drafting does not fix staffing.

**Checklist.** Store from the page the message arrived on. Triage writes tags and priority only. Drafts start from the approved macros and the learned procedures. Every fact in a draft is in its context. The lookup reports whether it failed. Actions are confirmed only after they run. Live sending needs consent outside the command line. Escalations get a holding reply and a human. Public drafts carry no personal data. Guards are tested in every language the drafts are written in.

## Questions and answers

### Should AI send customer service replies automatically?

We do not let it. Our language model writes a draft inside the helpdesk and a human agent reviews, edits and sends it. In an audit of 75 real drafts, 71% had problems, 18 of them serious, mostly invented facts such as delivery dates, prices or a claim that an order had been checked. Live sending exists in our code but refuses to run without consent given outside the command line.

### How do you stop an AI support draft from inventing order details?

Give the draft only real data, then check the final text. A draft may state a status, deadline, price or size only if that fact is in its context. A deterministic filter looks for invented lookups, statuses and deadlines after generation; a draft that trips it is regenerated once, and if it still invents, a safe template per category replaces it. Actions are confirmed only after they actually run.

### What are customer service macros and how should AI use them?

Macros are the support team's approved reply templates, one per ticket category and language, with variables such as customer name, order number, tracking number and return link. An AI draft should pick the macro for the ticket's category and language, keep its wording and format, and fill the variables with real data only, never with guesses.

### Why did our AI reply tell a customer they had no order?

Usually because the order lookup failed and the failure looked like an empty result. Our lookup returned an empty list when the database was unreachable, the same as a search with no matches. In a simulated outage on 162 real tickets, 83 drafts would have said the order did not exist. The lookup now reports three states, and a failed search forbids any statement about the order.

### Which support tickets should an AI never answer on the substance?

Legal or consumer-authority threats, refunds that were promised but not paid, very angry customers and repeat complaints. For these our engine saves only a short holding draft, raises the priority, tags the ticket for escalation and writes an internal note with the problem, the order and the tracking number, so a human handles the substance.

## How to cite

**Cite this as:** Going Stellar (2026). *How to use AI to draft customer service replies without letting it send them*. GOING STELLAR S.R.L.. https://goingstellar.ro/methods/ai-drafted-customer-service-replies/

```bibtex
@misc{goingstellar2026ai,
  title  = {How to use AI to draft customer service replies without letting it send them},
  author = {{Going Stellar}},
  year   = {2026},
  url    = {https://goingstellar.ro/methods/ai-drafted-customer-service-replies/},
  note   = {2026-10-11}
}
```

Licence. Free to quote with attribution.

## Also available

- This page as HTML: https://goingstellar.ro/methods/ai-drafted-customer-service-replies/
- Same article in Romanian: https://goingstellar.ro/ro/metode/raspunsuri-customer-service-scrise-de-ai/