How AI reduced manual support work by 40% for a 20-person SMB

A 20-person B2B SaaS company reached out to us in early March. Their support situation looked like this:

• One support lead, handling everything.
• 600+ Zendesk tickets per week, growing 15% month over month.
• A 200-page Notion workspace full of accurate, well-written documentation that customers never read.
• A founder who was about to “just hire two more support reps” and call it a day.

The founder did the math: two more support reps was about $130K all-in for the year. He told us he’d rather spend a quarter of that on an AI solution if we could make it actually work.

Four weeks later we shipped a GPT-4o-powered chatbot, embedded in their Intercom messenger, that now handles 4 out of every 10 incoming questions on its own. The other 6 get routed to the human support lead, but with a draft reply pre-written for him. The median customer response time dropped from 6 hours to 2 minutes. He never made those two hires.

This is how we did it — and what we’d do differently next time.

Why most “AI support bots” fail

We had to convince the founder not to do what most companies do, which is buy a generic chatbot platform, point it at their FAQ, and call it a day. Those projects fail for three reasons:

1. The bot doesn’t know your product. Generic chatbots train on your public docs, miss internal Notion pages, and have no awareness of which customer is asking.
2. The bot answers confidently when it shouldn’t. Hallucinations on technical questions destroy customer trust faster than a slow human response ever could.
3. The handoff to a human is broken. When the bot fails, customers have to re-explain everything. The agent has no context. The customer churns.

Our brief to ourselves was simple: build the bot a senior support lead would build for themselves. It should know everything the lead knows, refuse to answer when unsure, and hand off cleanly with full context when it does.

The architecture, in 200 words

• Knowledge source: The customer’s full Notion workspace, synced nightly via the Notion API. We deliberately didn’t include their old Zendesk tickets — too much outdated info, too much noise.
• Chunking: Each Notion page was split into 500-token chunks with 100-token overlap, with the page title and breadcrumb path preserved in metadata for every chunk.
• Embeddings: OpenAI text-embedding-3-small. Cheap, fast, accurate enough at this scale.
• Vector store: Pinecone, single index, ~40K vectors total. We didn’t need anything fancier.
• LLM: GPT-4o for response generation. We tested gpt-4o-mini and it was noticeably worse on the nuanced technical questions, so we ate the cost difference.
• Edge runtime: Cloudflare Workers. The bot needed to respond fast (sub-2 seconds end-to-end), and Workers gave us global edge latency with simple deployment.
• Front-end: We didn’t build one. The bot lives inside Intercom, using their Custom App API to inject responses into the existing chat UI.

Total infrastructure cost: about $480/month at their volume. Compared to $130K/year for two hires, the unit economics were obvious from week one.

The hard part: knowing when to shut up

The chatbot was answering questions correctly within the first week. The hard part was getting it to not answer when it shouldn’t.

We did three things that made the difference:

1. Hard confidence thresholds, not soft prompting

Every chunk retrieval came back with a similarity score. We set a hard threshold: if the top retrieved chunk had a score below 0.78, the bot wouldn’t respond. It would say: “I’m not sure about this one — let me get a human on it” and route to Intercom. No exceptions, no prompt-engineering trickery.

This single change dropped our hallucination rate from “embarrassing” to “we genuinely couldn’t find a case in the first 30 days.”

2. Citation-required answers

Every bot answer had to cite the source Notion page. The prompt made it non-negotiable: “If you cannot cite a specific Notion page, do not answer.” This had two effects: customers trusted answers more when they saw the source link, and the bot literally couldn’t make things up because the citation step would fail.

3. The “would a human be embarrassed?” review loop

For the first two weeks after launch, the support lead reviewed every bot response. Anything he wouldn’t have written himself got flagged, and we updated either the source documentation or the prompt. By week four, his review rate was below 5%, and he stopped reviewing entirely except for spot-checks.

What the human still does — and why we didn’t fire anyone

The 40% number is real, but it tells a misleading story. Here’s the more honest version:

• The bot handles 40% of tickets autonomously. The customer never talks to a human.
• For the other 60%, the bot drafts a reply that the support lead can send with one click or edit before sending. About 30% of those drafts go out unchanged. Another 20% need light edits. The remaining 10% are complex enough that the lead writes a full custom response.
• Net effect: the lead’s effective throughput is roughly 3× what it was before. Same headcount, much more output.

The founder was clear with us from day one: he didn’t want to replace his support lead. He wanted to give him superpowers. That framing changed how we built the product. We didn’t try to maximize “autonomous resolution rate” — we maximized time saved per ticket. Those are different optimization targets, and most AI support bots aim at the wrong one.

What we’d do differently

A few things we got wrong, and what we’d do next time:

We over-trusted the embedding model on technical content

text-embedding-3-small struggles with code snippets and very technical jargon. We had a few weeks where questions about a specific API endpoint kept retrieving unrelated marketing pages. We eventually solved it by adding a code-aware preprocessing step that extracted code blocks separately and indexed them with their surrounding context. Next time we’d build that on day one.

We didn’t think about analytics until week 3

We launched without proper observability. When the founder asked “how is it doing?” we had to scrape logs to answer. Now we ship every bot with a metrics dashboard from day one: resolution rate, citation accuracy, response time, refusal rate, customer satisfaction follow-up.

Customer feedback loops should be in-product, not in surveys

We initially planned to send weekly NPS surveys. Useless. The data we actually used came from a single line of follow-up text in the chat: “Did this answer your question? 👍 / 👎.” That feedback flowed straight back into our prompt-tuning queue, and it told us in real time which docs were stale.

If your team is drowning in repetitive support tickets, the math is usually obvious within the first 30 minutes of a conversation. Message us on WhatsApp — we’ll look at your ticket data with you and tell you whether AI helps or whether you just need better docs.