Catching an AI chatbot that regressed on every prompt change

The challenge

A live AI support chatbot regressed on every prompt change — a small wording tweak would quietly break answers that used to work, and nobody noticed until users complained.

The context

The team was iterating fast on prompts and swapping models, with no way to tell whether a change helped or hurt. Quality was a matter of opinion, and every release was a gamble.

What we did

• Built a golden set of representative questions and expected behaviours from real conversations.
• Graded answers for faithfulness, correctness and tone, flagging hallucinations and dropped context.
• Wired the evals into CI so every prompt or model change is scored before it can ship.
• Handed over the suite and the CI gate for the team to keep extending.

The outcome

Prompt and model changes are now scored automatically, and a regression blocks the release instead of reaching users. Quality moved from opinion to a number the team can trust.