Monica AI: The First RAG Product in My Category
Every competitor in my market had shipped AI by 2023, and all of it was cosmetic. Summarise a ticket. Rephrase a reply. Suggest a canned response. It demoed well and changed nothing. A technician's day looked identical the week after launch, and sustained use of these features sat in the single digits industry-wide.
I thought the real opportunity was the harder one underneath. A service team is sitting on years of resolution history: every problem it has ever solved, plus the knowledge base it has authored. If a product could retrieve from that and recommend the next action for the ticket in front of you, with enough trust that a technician would act on it under pressure, that would change the day. The bet: not a smarter chatbot, but a retrieval product aimed at the moment of work.
The binding constraint was trust. A technician who follows a wrong recommendation burns time, and on a billable ticket that's money walking out the door. So the product couldn't just be right on average. It had to show its reasoning well enough that a skeptical, experienced technician would lean on it.
Custom RAG over a wrapped general model, built to be explainable. The default path in 2023 was to wrap an LLM and call it AI. I rejected it: a wrapped model produces a fluent answer with no traceable basis, and in this workflow an unverifiable answer is worse than none, because it costs the technician the time to second-guess it. I anchored the product on retrieval from the customer's own data and made the evidence first-class.
Explainability sourced from both past tickets and the knowledge base. Every recommendation surfaces where it came from (the specific prior tickets the team resolved and the relevant KB articles they authored), so a technician can read why this answer and drill into the source before acting. Drawing on both is deliberate: past tickets carry how this team actually solved this class of problem; the KB carries the sanctioned procedure. Showing both is what earns trust from someone who knows the domain cold.
Target the segment where the value compounds. I aimed Monica at teams of 10+ technicians. Minutes saved per ticket only become money when there's enough volume for them to roll up; pointing the product there meant the same engineering effort produced a result a customer could feel on their P&L.
Design for a permanent iteration loop. Retrieval quality isn't a thing you get right once; it's tuned against real usage forever. I set the product up to test with live customers in days and feed results straight back into retrieval tuning.
A deliberate freshness/cost trade-off. Real-time-to-the-second retrieval would have been expensive and brittle; nothing about recommending the next action needs sub-minute latency. I accepted near-real-time freshness as the right balance: a conscious, documented call.
The architecture we designed with engineering followed a data-lake pattern: production data captured in near-real-time through a custom change-data-capture pipeline, streamed into an analytical store, with a vector database powering semantic retrieval over two corpora: the resolution history and the knowledge base.
What decided whether it worked was tuning what "relevant" means. We tested different vector-distance metrics and similarity thresholds against real usage, watching for the failure mode that quietly kills these products: returning something plausible but wrong, which erodes trust faster than returning nothing. Set the threshold too loose and the system surfaces weak matches that make a technician stop trusting it; too tight and it goes silent and feels useless. We tuned that boundary empirically (distance scoring, similar-ticket thresholds, KB-match thresholds) in a tight loop across engineering, design, and customer success: ship to a cohort, watch what technicians actually clicked into and acted on, adjust, repeat. The launch version and the version that hit 84% adoption were meaningfully different products; only the iteration loop got us from one to the other.
Established the product as the explainable-AI leader in the category, and built the data foundation and iteration loop that every later AI initiative ran on. Monica was the platform's AI on-ramp.
AI products win on workflow fit, explainability, and iteration speed, above model choice. Two failure modes are everywhere in B2B AI: skip the data foundation and you can't keep tuning, so quality plateaus and slides; hide the AI's reasoning and you lose trust the first time it's wrong, and there's always a first time. We designed against both: retrieval grounded in the team's own tickets and KB, the evidence shown, the thresholds tuned against real use. That is why it reached 84% and stayed there.
Built the data foundation and adoption analytics from day one, so retrieval could be tuned against real use and every recommendation showed its sources.
Read the thesis