# How Trieve resolves production issues in minutes with Subtrace

**Company:** Trieve  
**Website:** [https://trieve.ai](https://trieve.ai/)  
**Description:** Infrastructure for search teams building retrieval and RAG. Trieve combines search language models with tools for tuning ranking and relevance.  
**Industry:** Technology

Trieve's mission is to make AI search accurate, reliable, and fast. Founded in 2023, Trieve lets developers add a conversational search bar to their app in just minutes. Over 30,000 websites use Trieve to handle tens of millions of queries every month.

Engineers at Trieve use Subtrace every day to debug their backend services and resolve incidents quickly. After struggling to root cause issues with traditional observability platforms, they switched to Subtrace and improved their resolution time on revenue-critical incidents from months to just minutes. Trieve’s founders tell us how important Subtrace has been for them.

## Top Highlight

> “It's like living in utopia. I never want to build anything without Subtrace ever again. Life changing piece of software.”  
> — Nick Khami, CEO, Trieve(YC W24)

## Death by a thousand tiny outages

Searching millions of documents in milliseconds at scale is incredibly difficult. Before Subtrace, Trieve constantly struggled with outages and latency spikes in their core services. They tried a patchwork of observability tools to fix these problems, but every incident became a nightmare spanning five monitoring tools, twelve browser tabs, and zero answers to the questions that matter.

“Our first set of questions are the same every time we debug an API error or slow request: which tenant, which API route, what was the query payload, and what was the response,” says Nick Khami, Trieve's CEO. “Answering any one of those questions was an ordeal. It was incredibly difficult to see the actual details of any given request.”

After months of alerts that always howled *something's wrong* and never *here's why*, they decided it was time to look for alternatives.

## Finding the signal in the noise

When Denzell Ford, Trieve's CTO, opened Subtrace for the first time, he expected yet another dashboard. What he got was a flashlight.

Setup was just one command—no dependencies, no code changes, no complex permissions. Trieve's engineers could suddenly see every API request flowing through their systems in real time. “We instantly identified the root cause of a major latency spike to be a bad Postgres query,” says Denzell. “Using Subtrace, we also discovered errors in our service that we couldn't see anywhere else before.”

With Subtrace, they could replay exact requests, see headers, analyze latency breakdowns. No more guessing. Things they'd previously dismissed as random glitches suddenly had names, traces, timelines. Engineers could now fix the root cause, not the symptom.

## Trieve's conclusion: Subtrace makes them more reliable and lets them respond to customer issues faster

Recently, a customer reported an API outage via Slack. Before Subtrace, Trieve's engineers would've wasted hours sifting through disconnected logs across dozens of servers. This time, they used Subtrace to simply find the customer's exact request based on a HTTP header, then inspected its response status code and error message in the payload. It turned out that Kubernetes had failed to automatically restart an unhealthy pod. They manually restarted it, then used Subtrace again to verify that the fix actually worked for the customer by checking the status code of subsequent requests. It worked. The issue wasn't complex—just invisible. The whole incident was resolved faster than the customer's next Slack message.

This is just one example of how Trieve uses Subtrace to improve reliability and respond to customer incidents faster. “We now have the utmost confidence in communicating through issues with our customers,” says Nick.

Nick and Denzell also appreciate Subtrace's open source development. “Trieve is open source, so it makes sense that our tools would be too,” says Nick. Above all, they love Subtrace because it lets them identify the root cause of each incident in just minutes, leading to fewer outages, smoother operations, and happier customers.

> “It's like living in utopia,” Nick concluded. “I never want to build anything without Subtrace ever again. Life changing piece of software.”