Last month, our team hit a wall. We had over forty microservices running across three GKE clusters, and every time an on-call engineer got paged at 2 AM, they had to jump between Grafana, PagerDuty, our internal wiki, and three different CLI tools just to figure out what was going on. I kept thinking, "What if they could just ask a question and get the answer?"
That's what led me down the rabbit hole of Google's Vertex AI Agent Builder. I didn't want a generic chatbot—I wanted an agent that could actually do things in our infrastructure. Here's how I built a DevOps assistant that could query our systems and execute runbooks, and what I learned along the way.
The Problem: Context Switching is Killing Us
Our incident response looked like this: get alert, open dashboard, realize you need logs, switch to log viewer, realize you need to scale a deployment, open terminal, realize you forgot the cluster context, switch again. Five tools for one incident. I wanted to collapse that into a single conversational interface.
Vertex AI Agent Builder caught my attention because it lets you connect an agent to external tools via OpenAPI specs. Instead of hardcoding logic, I could point the agent at our existing APIs and let it figure out when to call them.
Step 1: Setting Up the Groundwork
First, I headed to the Google Cloud Console and navigated to Vertex AI Agent Builder. You'll be prompted to activate the API—click "Continue and Activate the API." It takes a minute or two.
Once activated, I created a new app. You get a few options here. For DevOps use cases, I recommend starting with a "Agent" type rather than the simpler "Search" type, because we need the agent to take actions, not just return documents.
I named mine devops-assistant-prod and selected our project's region. One thing that surprised me: the Agent Builder interface is separate from the main Vertex AI section in the console. You have to specifically navigate to "Agent Builder" under the Vertex AI navigation panel. I spent twenty minutes looking in the wrong place the first time.
Step 2: Defining the Agent's Behavior
The agent needs instructions—essentially a system prompt that tells it how to behave. Here's what I wrote:
You are a DevOps assistant for our production infrastructure.
You have access to tools that can query Kubernetes clusters, check monitoring dashboards, and execute runbooks.
When a user reports an issue:
1. First, gather information using the available tools.
2. Summarize the current state clearly.
3. Suggest remediation steps, and ask for confirmation before executing any state-changing operations.
4. NEVER restart pods or scale deployments without explicit user confirmation.
If you are unsure about a command or action, say so. Do not guess.
The key insight here: being extremely specific about safety constraints. You do not want an agent deciding on its own to scale a production cluster to zero because it misinterpreted a request. Trust me, I tested this without the guardrails and it immediately tried to restart all our canary deployments.
Step 3: Connecting DevOps Tools via OpenAPI
This is where the magic happens. Agent Builder lets you define tools using OpenAPI (Swagger) specifications. Our internal services already had OpenAPI docs, so I just needed to clean them up and point the agent at them.
Here's a simplified version of the OpenAPI spec I wrote for our Kubernetes query API:
openapi: 3.0.0
info:
title: DevOps Internal API
version: 1.0.0
paths:
/api/v1/pods:
get:
summary: List pods in a namespace
parameters:
- name: namespace
in: query
required: true
schema:
type: string
description: "The Kubernetes namespace to query"
- name: status
in: query
required: false
schema:
type: string
description: "Filter by pod status (Running, CrashLoopBackOff, Pending)"
responses:
'200':
description: List of pods
content:
application/json:
schema:
type: object
properties:
pods:
type: array
items:
type: object
properties:
name:
type: string
status:
type: string
restarts:
type: integer
/api/v1/scale:
post:
summary: Scale a deployment
requestBody:
required: true
content:
application/json:
schema:
type: object
required:
- namespace
- deployment
- replicas
properties:
namespace:
type: string
deployment:
type: string
replicas:
type: integer
minimum: 0
maximum: 20
responses:
'200':
description: Scaling initiated
In the Agent Builder console, I went to the "Tools" section, clicked "Add Tool," and uploaded this OpenAPI spec. I also configured the authentication to use our internal API keys stored in Google Secret Manager.
The important thing I learned: keep your OpenAPI specs tight. My first attempt included every endpoint from our entire API gateway—over 200 paths. The agent got confused and would call the wrong endpoints. Once I trimmed it down to just the 8 DevOps-relevant endpoints, the accuracy jumped dramatically.
Step 4: Adding a Knowledge Base for Runbooks
Tool execution is only half the battle. Our agents also needed access to our runbook documentation—those step-by-step guides for handling specific alerts.
In Agent Builder, I created a data store and pointed it at our Cloud Storage bucket where we keep Markdown runbooks. The setup was straightforward: create a new data store, select "Cloud Storage" as the source, point it at the bucket, and let it index.
One gotcha: the indexing took about 45 minutes for roughly 200 documents. And if you update a runbook, you need to trigger a re-import. It's not automatic. I set up a Cloud Build trigger to handle this, but it's an extra step you should plan for.
Step 5: Testing and Iterating
The Agent Builder console has a built-in preview panel. I spent a solid day testing different scenarios:
Test 1 - Information Query:
Me: "What pods are in CrashLoopBackOff in the payments namespace?"
Agent: Called the/api/v1/pods?namespace=payments&status=CrashLoopBackOffendpoint, then returned: "There are 2 pods in CrashLoopBackOff in the payments namespace:payment-processor-7d8f9(12 restarts) andpayment-validator-3a2b1(5 restarts). According to the Payment Service Runbook, this is typically caused by a failed database migration. Would you like me to check the migration status?"
This worked great. It combined the API call with the runbook knowledge.
Test 2 - Action Execution:
Me: "Scale the payment-processor deployment to 5 replicas."
Agent: "I will scale thepayment-processordeployment in thepaymentsnamespace to 5 replicas. Please confirm: yes or no?"
Me: "yes"
Agent: Called/api/v1/scalewith the appropriate payload. "Scaling initiated. The payment-processor deployment is now scaling to 5 replicas."
The confirmation step I built into the instructions worked exactly as intended.
Test 3 - The Edge Case That Went Wrong:
Me: "Fix the payments service."
Agent: Immediately called the scale endpoint to scale to 0, then back to 3.
This was terrifying. The agent decided "fix" meant "restart everything." I had to add much more specific instructions about never performing restart sequences without walking through diagnostic steps first. This is where you realize how important prompt engineering is for agents with real power.
Step 6: Deployment and Integration
Once I was confident in the agent's behavior, I deployed it. Agent Builder gives you a REST endpoint and integrates with Dialogflow for conversational interfaces.
I wired it up to our Slack workspace using a simple Cloud Function that bridges Slack events to the agent's API. The flow looks like this: someone types @devops-bot pods crashing in payments in Slack, the Cloud Function forwards it to the agent, the agent executes the tools and queries the knowledge base, and the response gets posted back to the thread.
Practical Tips and Honest Assessment
What worked well:
- The OpenAPI integration is genuinely useful. If you already have API specs, connecting them is fast.
- The knowledge base for runbooks saves a ton of context-switching time.
- The confirmation flow for destructive actions is reliable once you tune the instructions.
Limitations to be aware of:
- The agent sometimes hallucinates API parameters that don't exist. Always validate on your backend.
- There's no built-in rate limiting on tool calls. If the agent gets stuck in a loop, it can hammer your APIs. I had to add rate limiting on our API gateway side.
- Cold starts are real. The first query after a period of inactivity can take 5-10 seconds. Not great for urgent incidents.
- The debugging experience is rough. When the agent calls the wrong tool, figuring out why it made that decision involves digging through verbose logs that aren't easy to parse.
- Cost can sneak up on you. Every tool call and knowledge base query has a cost. During testing, I racked up a surprising bill before setting budget alerts.
My biggest takeaway: Start small. I initially tried to connect every DevOps tool we had. It was a mess. Now we have three focused agents—one for Kubernetes queries, one for incident runbooks, and one for deployment status. They each do one thing well, and the team actually trusts them because they're predictable.
Vertex AI Agent Builder isn't a replacement for your existing DevOps tools. It's a layer on top that makes them more accessible. If you go in with that expectation and invest time in tight OpenAPI specs and clear instructions, you can build something that genuinely saves time during incidents. Just don't skip the guardrails.