1. Pick the sample
Take the 50 most recent tickets your team answered. Use recent ones so the AI is judged on your current products and policies. Do not cherry-pick: include the dull ones and the awkward ones, because both are what the AI will see.
2. Give the AI what a new hire would get
The AI needs the same sources you would give a new colleague: your help articles, your macros, and the facts that are not written down anywhere (shipping times, refund rules, plan limits). Weak answers often trace to missing facts, not a weak model.
3. Score each answer
For every ticket, put the AI's draft next to what your team sent, and mark one of four:
- Send as is: correct, complete, and the right tone.
- Send with a small edit: right idea, one fix.
- Wrong: factually off, or promises something you do not do. This is the number that matters most.
- Should have been a person: a refund dispute, an angry customer, a bug report. Did the AI recognise that and hand it over?
4. Read the result as a rate and as a risk
If 20 of 50 are send-as-is and 8 are with small edits, the AI could plausibly finish somewhere around 40% on its own and speed up another 15% or so. That is your expected AI share for the bill estimate. Then look at the wrong ones: one wrong answer that promises a refund you do not give costs more than ten right answers save. Decide what the AI should never answer alone, and see whether it can be told.
5. Check the safety rails
Before you let it reply to real customers:
- Can you turn it off in one place?
- Does it say when it is unsure, and hand off with a note?
- Can a customer ask for a person at any time?
- Is there a limit on what it can spend, refund or change?
- Can you see every answer it sent and refund a wrong one?
A shortcut
Flatdesk's AI test drive does steps 1 to 3 for you: after you import your help desk it drafts replies to your 50 most recent answered tickets next to your team's replies, sends nothing, and uses none of your AI allowance. The method above works for any vendor.