Last month, our team deployed a customer support agent to production. It worked great in testing—answered questions accurately, pulled the right data, and responded in under two seconds. Then the tickets started rolling in. Users were getting weird, truncated responses. Some requests were timing out. Our traditional monitoring showed green across the board: CPU was fine, memory was fine, the API was returning 200s. But the agent was clearly broken somewhere inside its chain of tool calls and LLM requests.
That's when I realized our standard DevOps stack was blind to what actually matters in an LLM application. We had infrastructure observability, but zero application-level observability into the agent's actual decision-making process. I started looking for a solution and landed on LangSmith. Here's how I set it up and what I learned using it as a DevOps tool for our LLM workloads.
Setting Up Tracing: The Foundation
The first thing you need with LangSmith is tracing. This is the core capability—every request your agent handles gets recorded as a trace, which is a nested tree of all the individual operations (LLM calls, tool executions, retriever lookups, etc.).
Setting it up was surprisingly simple. After creating an account at smith.langchain.com and generating an API key from Settings → API Keys → Create API Key, I just needed to set environment variables:
export LANGSMITH_API_KEY="your-api-key-here"
export LANGSMITH_TRACING="true"
export LANGSMITH_PROJECT="prod-support-agent"
That's it. No code changes required if you're using LangChain or LangGraph—the SDK automatically picks up these environment variables and starts shipping traces. I was genuinely surprised at how little friction there was to get the first traces flowing.
For our FastAPI service, I added these to the container environment in our docker-compose file. Within minutes of redeploying, traces were showing up in the LangSmith UI.
What Traces Actually Show You
This is where LangSmith earned its keep. A single user request to our support agent triggered a sequence that looked like this in the trace view:
- Input parsing (2ms)
- Retriever call to fetch relevant docs (340ms)
- First LLM call to classify the query (1.2s)
- Tool call to look up order status (890ms)
- Second LLM call to format the response (980ms)
- Output validation (5ms)
Total: about 3.4 seconds. But here's the thing—when users reported timeouts, I could see exactly where the delay was. In one trace, the retriever call took 8 seconds because the vector store was throttling. In another, the second LLM call took 4 seconds because the model was under heavy load. The trace view shows you the full execution path with timing for each step, the exact inputs and outputs at every stage, and any errors that occurred.
For the truncated responses, I found the issue in minutes: our output validation step was silently cutting off responses that exceeded a token limit we'd set too aggressively. The LLM was generating complete answers, but our validator was chopping them. Without seeing the actual inputs and outputs at each step, I would never have found this.
Building Dashboards for Production Monitoring
Once tracing was running, I moved to setting up proper monitoring dashboards. LangSmith lets you build custom dashboards that track metrics across your traces. I set up panels for:
- Average latency per request (and p50/p95/p99 breakdowns)
- Error rate by component (retriever failures vs. LLM failures vs. tool failures)
- Token usage and cost tracking (this was an eye-opener)
- Tool call success rates
The cost tracking alone justified the tool. We were burning through tokens much faster than expected on our classification step because the prompt was unnecessarily verbose. I trimmed it down and cut that step's token usage by 40%.
You can also set up alerts. I configured a PagerDuty alert for when error rates exceed 5% over a 10-minute window, and a Slack webhook for when p95 latency crosses 5 seconds. These aren't just infrastructure alerts—they're alerts on the actual quality and performance of your agent's behavior.
Using Automations for Operational Workflows
One of the more powerful DevOps features is LangSmith's automations. You can define rules that trigger actions based on trace data. Here's what I set up:
Rule 1: Auto-flag long traces. Any trace exceeding 10 seconds gets tagged with "performance-review" and added to an annotation queue. This lets me review slow requests without manually hunting for them.
Rule 2: Error routing. Any trace with an error gets sent to a specific annotation queue and triggers a webhook to our incident channel. This replaced a janky custom error handler we'd built.
Rule 3: Online evaluations. LangSmith supports running LLM-as-judge evaluations on production traces automatically. I set up an eval that scores response completeness on a 1-5 scale. Any trace scoring below 3 gets flagged for review. This gives us a continuous quality signal without manual review of every output.
Setting these up through the UI was straightforward—pick a trigger condition, define the action, and enable it. No custom code needed.
Insights: Finding Patterns You Didn't Know Existed
The Insights feature was a pleasant surprise. It automatically clusters your traces to find patterns—common usage topics, recurring failure modes, unexpected agent behaviors.
In our case, Insights identified that about 15% of our support requests were following an unusual trajectory: the agent would call the order lookup tool, get a result, then call it again with slightly different parameters. It turned out our prompt was ambiguous about how to handle partial order numbers, causing the agent to retry with variations. This was a behavior I never would have spotted manually, and it was wasting tokens and adding latency on a significant chunk of requests.
The SmithDB Advantage for Querying Traces
One practical detail that matters at scale: LangSmith uses a purpose-built database called SmithDB for storing and querying traces. Agent traces are deeply nested—a single conversation can generate megabytes of data across dozens of runs and tool calls. I tried querying our trace data with standard tools first, and the JSON payloads made it painful. SmithDB supports JSON key-path filtering, full-text search, and trajectory queries (finding traces where the agent followed a specific path of actions). Queries run in sub-second time across millions of traces, which matters when you're debugging an incident and need answers now.
Practical Tips and Honest Limitations
Tips:
- Use separate projects for dev, staging, and prod. The
LANGSMITH_PROJECTenv var makes this easy. - Don't skip the annotation queues. Having a structured way to review and tag problematic outputs creates a feedback loop that improves your agent over time.
- Set up online evaluations early. Even a simple "is the response relevant?" eval running on 10% of production traces gives you a quality signal you'd otherwise lack.
- The environment variable approach means you can toggle tracing on/off without code changes—useful for debugging specific environments.
Limitations:
- LangSmith is purpose-built for LLM/agent observability. It doesn't replace your infrastructure monitoring (Datadog, Grafana, etc.). You need both.
- The free tier has trace limits that you'll burn through quickly in production. Budget for the paid tier if you're running real workloads.
- While it supports OpenTelemetry and multiple SDKs (Python, TypeScript, Go, Java), the deepest integration is with the LangChain/LangGraph ecosystem. If you're using a completely custom stack, you'll need to do more manual instrumentation.
- Trace data can get large. Be intentional about what you log—don't dump entire document contents into trace metadata unless you actually need it.
LangSmith filled a critical gap in our DevOps stack: visibility into what our agent was actually doing. Traditional monitoring told us our servers were healthy. LangSmith told us our agent was making bad decisions. For any team running LLM applications in production, that distinction is the difference between thinking everything's fine and actually knowing it is.