What a mock didn’t tell me: putting my profile chat on a real model
Every test passed on a mock model. On the real one, 40% of answers fell back to an emergency mode. Four product decisions that fixed it, and what you can take from them.
The chat on this site answers questions about my work. Its only source is my profile: a closed list of statements about my career, each backed by a verbatim quote from my own case studies. The one hard rule is no facts from outside the profile. AI agents wrote the code, the tests and the measurements. I set the architecture, the rules and the quality thresholds, and made every call on what ships. This post is about what happened when the chat met a real model instead of a stand-in.
All tests green, 40% of answers broken
The first version split the work between three AI roles. One decides what kind of question it is and which part of the profile matters. A strong model writes the answer. A faster, cheaper model checks it. We tested it against a mock model with canned answers, and every test passed.
On the real model the chat stayed safe. It never stated a fact from outside the profile, it held every boundary I set, and it said "I don't know" when it should. Yet about 40% of answers fell back to an emergency mode: when an answer fails the checks, the chat shows the matching quotes from the profile instead, so it never says something unsupported. A typical question took around 11 seconds.
The loop cost more than the model
A single call to the strong model took about 2.5 seconds. The time went into a loop: write, check, rewrite. The checker saw less context than the writer, so it rejected true statements such as "5.5 years at ING" as unsupported, and every rejection meant writing the whole answer again. In 11 of 16 questions I profiled, the answer was written twice.
A second cause showed up later. In 12 of 40 calls the model ignored the answer format we required, or wrote long lists that got cut off. The mock always returned short, well-formed answers, so neither problem could appear in testing.
Four decisions
Measure only where the chat runs. Quality, time and cost are measured only on the production machine, never on a laptop. Each round asks 30 questions in English and Polish, each paid run has a spend limit, and each experiment ends with a written decision. The trade-off: a round takes longer to set up and costs about 0.7 USD.
Pin down the answer format. Hard limits on length and structure, enforced in code and re-tested on the live model before every measurement. The trade-off: shorter answers, so broad questions get a summary.
Take the model checker off the user's path. While someone waits, fixed rules check each sentence: the names and claims in it must match the profile. That works because the profile is a closed list. The AI judge still reads answers, but only in our measurement rounds. The trade-off: rules are stricter than a model, so now and then a correct sentence gets dropped.
Show sentences as soon as they pass the rules. The trade-off: about a quarter of questions still need a correction round, and those answers, along with hypothetical questions, arrive in one piece.
What it changed
Pinning down the format cut emergency answers from about 40% to 7%. Moving the checker off the path brought them to 3%. Among answers to questions about my experience, the share our AI judge rated as sensible went from 55% to 96%. With streaming on it settled around 91%, a dip we accepted in exchange for the first sentence arriving early, and the next rounds track it. A typical full answer now takes about 3 seconds instead of 11, and half of all answers start appearing within 1.6 seconds. In no round did the chat state a fact from outside the profile.
What you can take from it
- A mock tests the plumbing. If your product depends on how a model phrases and formats its answers, test it on the real model early, in the environment where it will run.
- Record time and cost for every step separately. The loop was invisible until each call had its own number.
- A cheaper model checking a stronger one can cost more than it saves. If your source of truth is a closed set, put fixed rules on the user's path and keep model judgement for offline evaluation.
- Perceived speed is a product decision. Showing verified parts early changed how the chat feels more than any single speed fix.
Next on the list: Polish word forms when the chat looks things up in the profile, and questions that span several places I worked at.