How I Built This Site's Agent and Tuned It With Evals

The agent on this site answers questions about my work, grounded in the site's own pages and citing the ones it draws from. This is an account of how it is built: the grounding, a failure I found and fixed during testing, how the evaluations are calibrated, and how the site measures whether any of it works. I built it over a weekend. The harder part was making it trustworthy.

Why there is no vector database

The common pattern for an agent like this is a vector database: split the content into chunks, embed them, and retrieve the closest matches for each question. I decided against it. The corpus is the published site, about sixteen thousand tokens, and it fits in the model's context in full. A vector database is worth the added complexity when the content is too large to fit and you have to select which parts to include. Here it would only add a failure mode, a retrieval step that occasionally fetches the wrong chunk and makes the agent miss something that was present all along. Matching the approach to the size of the problem mattered more than reaching for the more elaborate option. I have written elsewhere about deciding what not to build →, and this was a small case of it.

Grounding and citations

The agent is grounded on a single source: the content files that render the pages you can read. It has no access to my private notes, to anything marked confidential, or to internal strategy documents. This is a deliberate security decision. An agent cannot disclose information it was never given, so rather than rely on a prompt to keep quiet about sensitive material, I kept that material out of its context entirely. Each section of the grounding text also carries the route of the page it came from, so every claim the agent makes cites a page you can open and check. It cannot cite a source that does not exist, because the only sources it has are real pages. For a stranger deciding whether to trust it, grounding and citation together are what make the output verifiable rather than merely fluent.

A failure in testing

During testing I asked the agent how I handle failure, expecting it to draw on the Smart Tracker case study →, a detailed account of a launch that stalled and how I recovered it. Instead it returned the generic fallback message with my contact details. An off-topic question was declined correctly, but a straightforward, answerable one was failing.

My first assumption was a citation-slug mismatch from a page I had recently renamed. It was wrong, and the traces showed it. Every call is instrumented with full observability, the same approach the AI-Native Service Desk → case describes, so I could read what happened rather than guess. The cause was that the model provider was shedding large-context requests under load: small prompts usually succeeded, but the full sixteen-thousand-token request failed on eight consecutive attempts, and the occasional success took eleven to twenty-five seconds. I was compounding it myself, because my request timeout was set to around nine seconds and was cutting off valid answers that took longer.

The fixes were straightforward once the cause was clear. I raised the timeout, added retry with backoff on transient failures, and enabled context caching, which the corpus qualifies for on size, so each question now sends only the new tokens instead of resending the whole corpus. I did not need a vector database to reduce the request; I needed to stop resending content that had not changed. I also settled the cost question with data: use the free options first, measure, and pay for a backup provider only if the traces still showed a problem. They did not.

Calibrating the evaluations

Before the agent went live I wrote an evaluation suite, including a category of adversarial tests that probe for information it must never disclose: the name of an unreleased product, internal pricing, an acquisition target, and similar. The agent passes these only by declining, and the suite is a hard gate. It does not ship unless it scores perfectly on that category on every run.

The first run scored six out of fourteen. On the surface it looked as though the agent was disclosing the things it was meant to protect. Reading the transcripts showed the opposite: it had declined cleanly every time, and the grader was miscalibrated. I had defined a leak too loosely, so the judge flagged clean refusals, treated the agent stating its own public role as a disclosure, and read a neutral line about being open to a conversation as a signal it should not have inferred. The agent was behaving correctly. The measurement was not