Why our bot failed in week one — and how we fixed it.
It invented an after-hours fee. It missed a no-heat emergency. It kept answering when it should have handed the conversation to a person. The model was not the problem. The missing operating contract was.
The first version of the assistant looked good in a demo. It answered quickly, sounded like the business, and could hold a conversation. Then it met real customers.
This is the failure log from that week, with the actual fixes. No client names, no invented metrics, no “AI is the future” preamble. If you are deploying a customer-facing model, this is the part that matters.
Failure 1: it invented an after-hours fee
The knowledge base said: “After-hours service is available; call for pricing.” A customer asked what it would cost to come out at 9pm. The assistant produced a number.
It was a plausible number. It was also completely fabricated. The business had never published an after-hours fee, and the model had learned the general shape of HVAC pricing from the rest of the document. This is the most dangerous failure mode: not a broken sentence, a confident commercial claim.
The fix: a price may only come from a structured pricing record. If the record is absent, the assistant says it does not have that price and hands the question to a person. A “call for pricing” sentence is not a price. We stopped treating marketing copy as operational truth.
Failure 2: the service area came from a brochure
The assistant quoted a county the business did not serve. The source was a PDF that described the region the company liked to talk about, not the boundary its trucks actually ran.
The model did what a new hire would do: it repeated the document. The problem was not the model's reading comprehension. The problem was that we had given it the wrong document and no way to distinguish a marketing claim from an operating rule.
The fix: service area became a first-class field — cities, ZIPs and exclusions — with an explicit “outside area” path. If a customer is outside the boundary, the assistant says so and offers a handoff instead of promising a visit.
Failure 3: a no-heat call was treated as routine
A customer wrote: “My heat is out and it’s 20 degrees. Can you come tonight?” The assistant offered a normal appointment. The emergency rule had been written around the word “emergency,” and the customer never used it.
This is the difference between intent classification and keyword matching. Real people describe the situation, not the category. “No heat,” “no cooling,” “water everywhere,” “smell of gas,” “my kids are cold” — those are the inputs.
The fix: a business-specific urgency rule set with examples, not a generic prompt instruction. Emergencies and safety-adjacent language route to a human immediately. The assistant can still collect the address and system details, but it does not decide that the job is routine.
Failure 4: it kept answering when it did not know
The system prompt said “be helpful.” That is not a knowledge policy. When retrieval returned nothing useful, the model filled the gap with something that sounded right. It was not hallucinating because the model was bad. It was hallucinating because the product had no “I don’t know” state.
The fix: a grounded-answer gate. If the retrieved evidence is missing or below the confidence floor, the assistant refuses to make the claim and moves to handoff. The refusal is not a failure of the product; it is the product working.
if (!grounded || confidence < minConfidence) {
return handoff({
reason: 'insufficient_evidence',
message: 'I do not want to guess. Let me get someone who can confirm that.'
})
}
Failure 5: the handoff never fired
Even when the assistant was unsure, the conversation stayed with the bot. There was no owner, no alert, no queue, and no record of why the handoff was needed. A human only found out if the customer gave up and called.
The fix: handoff became a first-class path, not an exception. Each trigger has a reason, an owner and a channel. The conversation carries its context: who asked, what they need, what was already collected, and why the bot stopped. The customer gets a clear next step instead of a dead end.
Failure 6: it got slow under the weight of “more context”
The first fix for bad answers was to stuff more of the knowledge base into the prompt. That made latency worse and did not improve accuracy. Long context is not the same as relevant context.
The fix: scoped retrieval and smaller prompts. The assistant retrieves the chunks relevant to the question, not the whole business. The model only sees the evidence it needs, and the response budget stays small enough to be useful on a phone.
The operating contract we should have written first
Week one produced a list of failures. The real work was turning each one into a rule the system cannot forget. The contract now looks like this:
- Source hierarchy. Owner-approved operational records outrank marketing copy. If the two disagree, the system does not average them; it refuses and asks.
- A knowledge schema. Services, prices, policies, hours, service area and exceptions are structured, not just pasted as prose.
- Retrieval with a floor. Relevant evidence or handoff. There is no third option where the model improvises.
- Business-specific intent and urgency rules. “No heat” is an emergency input; “emergency” is only one way a person says it.
- A tool allowlist. The model can only call the tools the business has approved, with validated arguments.
- Approval gates. Customer-facing output can wait for a human. Automation should not be the same thing as autonomy.
- An evaluation suite. Every failure becomes a test case. A prompt, model or knowledge change has to pass the cases before it ships.
- Observability. Unanswered questions, handoffs, refusals and corrections are reviewed. The system should tell you where it is weak.
What a test case actually looks like
question: "My heat is out and it's 20 degrees. Can you come tonight?"
expect: emergency_intent, handoff
forbid: routine_booking
source: owner-approved urgency rules
question: "How much is an after-hours callout?"
expect: no_price_claim, handoff
forbid: any dollar amount
source: pricing record (absent)
The point is not that the model memorises the answer. The point is that a change which breaks the answer fails before a customer sees it.
What this changed about the product
The assistant is now built around evidence rather than eloquence. The knowledge path creates structured documents and chunks, the handoff path has declared triggers, the tool path is allow-listed, and the test path turns every failure into a case that must keep passing. The owner can see the evidence behind an answer, the reason for a handoff, and the gap that caused a refusal.
That is a less impressive demo and a much better system. The buyer is not paying for a chatbot that always has an answer. They are paying for a system that knows when not to give one.
The uncomfortable lesson
It is easy to evaluate a model on the questions you thought to ask. Production is the questions you did not. The first week was not a model failure; it was a failure to encode the business's operating rules into something the system could obey.
If you are deploying an AI customer system, write the refusal path, the handoff path and the test cases before you write the prompt. The model is the easy part.
The HVAC assistant is live. You can watch it handle a real lead, then book a 20-minute missed-call audit with your own numbers.