Last month, I deployed a customer support chatbot that I thought was working fine. Users were asking questions, it was giving answers, and my basic error logging showed zero crashes. Then I started manually reviewing conversations and realized something was off: the bot was confidently giving wrong answers about 20% of the time, it was taking 15+ seconds on certain queries, and it was occasionally hallucinating policies that didn't exist. My error logs were clean because the app wasn't crashing—it was just quietly failing at its actual job.
That's when I realized I needed actual observability into my LLM application, not just infrastructure monitoring. I needed to see what the model was thinking, how long retrieval was taking, and where the chain was going off the rails. After some research, I landed on LangSmith. Here's how I got it working and what I learned along the way.
The Problem: Flying Blind with LLM Apps
If you've built anything with LLMs, you know the feeling. You write a prompt, test it a few times in your notebook, and it looks great. You deploy it, and suddenly you have no idea what's happening. Traditional APM tools like Datadog can tell you that an API call happened and how long it took, but they can't tell you:
- What prompt was actually sent to the model
- How many tokens were consumed
- What the retrieval step returned before the model generated its answer
- Where in a multi-step chain things went sideways
LangSmith solves this by giving you trace-level visibility into every step of your LLM application. Let me walk you through setting it up.
Step 1: Account and API Key Setup
Head to smith.langchain.com and sign up. No credit card required, which I appreciated—you can actually kick the tires before committing. I signed up with my GitHub account, which took about 10 seconds.
Once you're in, you need an API key. Go to Settings → API Keys → Create API Key. Copy it immediately because you won't see it again. I made the mistake of not saving mine and had to regenerate it. Not a big deal, but annoying.
Step 2: Wiring Up Environment Variables
This is where I expected complexity, but LangSmith keeps it simple. You just set environment variables, and tracing starts automatically if you're using LangChain. Here's what you need:
export LANGSMITH_API_KEY="your-api-key-here"
export LANGSMITH_TRACING="true"
export LANGSMITH_PROJECT="my-customer-support-bot"
The LANGSMITH_PROJECT variable is optional but important. It groups your traces into a project so you're not looking at a giant undifferentiated mess. I wish I had set this up from day one instead of dumping everything into the default project and having to sort it out later.
If you're using a .env file (which you should be):
LANGSMITH_API_KEY=lsv2_pt_your_key_here
LANGSMITH_TRACING=true
LANGSMITH_PROJECT=customer-support-bot
Step 3: Seeing Your First Traces
Here's the part that genuinely surprised me. If you're using LangChain, you don't need to change a single line of application code. The environment variables hook into LangChain's callback system automatically. I ran my existing RAG chain:
from langchain_openai import ChatOpenAI
from langchain.chains import RetrievalQA
llm = ChatOpenAI(model="gpt-4", temperature=0)
chain = RetrievalQA.from_chain_type(llm, retriever=my_retriever)
result = chain.invoke("What is your return policy for electronics?")
And boom—when I checked the LangSmith UI, there was a complete trace showing every step: the initial query, the retrieval step with the actual documents fetched, the prompt sent to GPT-4, and the final response. I could see exactly which documents were retrieved and how long each step took.
That first "aha" moment was seeing that my retriever was returning irrelevant documents for about 30% of queries. The model was actually doing a decent job with bad inputs, but the retrieval was the weak link. I never would have caught that without trace-level visibility.
Step 4: Using LangSmith Without LangChain
Here's something the docs don't emphasize enough: you don't need LangChain to use LangSmith. I have some code that calls the OpenAI SDK directly, and I wanted tracing there too. You can use the LangSmith SDK directly:
from langsmith import traceable
from openai import OpenAI
client = OpenAI()
@traceable(name="generate_response")
def generate_response(user_query: str):
response = client.chat.completions.create(
model="gpt-4",
messages=[
{"role": "system", "content": "You are a helpful support agent."},
{"role": "user", "content": user_query}
]
)
return response.choices[0].message.content
result = generate_response("What is your return policy?")
The @traceable decorator is all you need. It logs the inputs, outputs, and timing to LangSmith automatically. You can nest these too—so if you have a multi-step pipeline, each step gets its own trace within the parent run:
@traceable(name="retrieve_docs")
def retrieve_docs(query: str):
# Your retrieval logic here
return documents
@traceable(name="generate_response")
def generate_response(query: str, docs: list):
# Your generation logic here
return response
@traceable(name="full_pipeline")
def rag_pipeline(query: str):
docs = retrieve_docs(query)
response = generate_response(query, docs)
return response
This gives you the same nested trace view in the UI, showing exactly how data flows through your pipeline.
Step 5: Actually Investigating Problems
Once tracing was set up, I used LangSmith to diagnose my chatbot issues. Here's what I found by filtering traces in the UI:
Slow queries: I sorted by latency and found that queries taking over 10 seconds all had one thing in common—the retriever was returning too many documents (I had it set to top-k=10), and GPT-4 was processing all of them. Dropping to top-k=4 cut latency by 60% with no quality loss.
Hallucinations: I filtered for traces where the model mentioned specific policy details and compared the retrieved docs. In several cases, the docs didn't contain the information the model was citing. The model was filling gaps with plausible-sounding fiction. Fix: I added explicit instructions to the prompt: "If the retrieved documents don't contain the answer, say you don't know."
Token waste: The trace view shows token counts per step. I realized my system prompt was 800 tokens of boilerplate that could be cut to 300. Across thousands of requests, that adds up fast.
Step 6: Setting Up Monitoring and Alerts
After fixing the immediate issues, I wanted to catch problems proactively. LangSmith lets you build dashboards and set up automations. I created a simple rule: alert me if the average latency for my project exceeds 5 seconds over a 10-minute window, or if the error rate goes above 5%.
You can also set up online evaluations that score traces automatically—like checking if responses contain certain keywords or running a cheaper model to rate response quality. I haven't gone deep on this yet, but it's on my list.
Practical Tips and Honest Limitations
Tips:
- Set
LANGSMITH_PROJECTfrom the start. Separate projects for dev, staging, and production. Future you will be grateful. - Use the
@traceabledecorator even for non-LLM steps like database lookups or API calls. Seeing the full pipeline is where the real value is. - The trace comparison feature is underrated. Select two traces side by side to see exactly why one worked and one didn't.
- Tag your traces with metadata like
user_tierorquery_type—it makes filtering much more powerful later.
Limitations:
- The free tier has trace limits. If you're running high-volume production traffic, you'll hit them faster than you think. I burned through my free allocation in about two days of testing.
- The UI can be sluggish when loading traces with very long prompts or large retrieved document sets. Patience required.
- If you're not using LangChain, the
@traceabledecorator works well, but you lose some of the automatic nesting that LangChain provides. You'll need to be more intentional about how you structure your traceable functions. - The documentation is still catching up in places. I had to dig into the SDK source code a few times to figure out advanced configuration options.
Bottom line: LangSmith fills a gap that traditional monitoring tools can't. If you're running LLM applications in production, you need this kind of visibility. The setup is surprisingly painless, and the insights are immediate. Just be mindful of the pricing as you scale, and structure your projects and tags thoughtfully from the start.