Last quarter, our DevOps team hit a wall. We had three different monitoring systems, two ticketing platforms, and a messy web of Lambda functions trying to glue them together. When a critical service degraded at 2 AM, our on-call engineer spent 45 minutes just switching between Grafana, PagerDuty, and ServiceNow to piece together what was happening. By the time they correlated the logs and triggered the runbook, the blast radius had already spread.
We needed a way to let our automation actually talk to each other—something that could take a signal from monitoring, look up the remediation steps, and execute them across our tools. That search led me to IBM watsonx Orchestrate. I was skeptical at first—"agentic control plane" sounds like marketing fluff—but after spending a few weeks building DevOps workflows with it, I found a genuinely useful tool for connecting fragmented systems. Here's how I got it working.
The Core Idea: An Orchestrator for Your Existing Tools
watsonx Orchestrate isn't another monitoring tool or CI/CD platform. Think of it as a routing and coordination layer that sits on top of the tools you already use. It connects to ServiceNow, Jira, Salesforce, AWS, and custom internal systems via OpenAPI specifications, then uses an orchestration engine to coordinate tasks across them.
For DevOps, this means instead of writing brittle custom scripts to bridge your alerting and remediation systems, you define agents that can interact with your tools and coordinate workflows end-to-end.
Getting Started: Setting Up the Agent Development Kit
While watsonx Orchestrate has a no-code visual builder (great for letting ops teams tweak workflows without touching code), I wanted programmatic control. I started with the Agent Development Kit (ADK), which lets you build and deploy agents as code using Python.
First, I installed the ADK via pip:
pip install watsonx-orchestrate-adk
The ADK gives you a CLI for creating, evaluating, and deploying agents directly. I initialized a new agent project:
orchestrate agents init devops-remediation --template python
cd devops-remediation
This scaffolds a basic project structure with an agent.yaml configuration file and a Python module where you define your agent's logic.
Building a DevOps Remediation Agent
My first goal was simple: build an agent that receives a PagerDuty alert, looks up the service in our CMDB, checks recent deployments, and—if a bad deploy is suspected—triggers a rollback in ArgoCD.
Step 1: Define the Agent Configuration
In agent.yaml, I defined the agent's metadata and the tools it could access:
name: devops-remediation
description: Handles incident triage and automated remediation for production issues
tools:
- name: pagerduty_lookup
openapi_spec: ./specs/pagerduty.yaml
- name: servicenow_cmdb
openapi_spec: ./specs/servicenow.yaml
- name: argocd_rollback
openapi_spec: ./specs/argocd.yaml
- name: github_commits
openapi_spec: ./specs/github.yaml
The key here is that watsonx Orchestrate consumes standard OpenAPI specifications. I didn't have to write custom integrations—I just pointed it at the existing API specs for our tools. For internal services, I wrote quick OpenAPI specs by hand.
Step 2: Write the Agent Logic
In the main Python file, I defined the agent's behavior using the ADK's framework:
from orchestrate import Agent, Tool, step
class IncidentRemediationAgent(Agent):
name = "incident_remediation"
@step
def triage_alert(self, alert_id: str):
"""Fetch alert details and identify affected service"""
alert = self.tools.pagerduty_lookup(alert_id=alert_id)
service_name = alert["service"]["name"]
cmdb_record = self.tools.servicenow_cmdb(
action="query",
service=service_name
)
return {"alert": alert, "cmdb": cmdb_record}
@step
def check_recent_deploys(self, cmdb: dict):
"""Check if a recent deployment correlates with the incident"""
repo = cmdb["git_repository"]
recent = self.tools.github_commits(
repo=repo,
hours=4
)
if recent["deploy_count"] > 0:
return {"suspect_deploy": True, "details": recent}
return {"suspect_deploy": False}
@step
def execute_rollback(self, deploy_info: dict):
"""Trigger rollback in ArgoCD if bad deploy detected"""
if deploy_info.get("suspect_deploy"):
result = self.tools.argocd_rollback(
app_name=deploy_info["details"]["app_name"],
revision=deploy_info["details"]["previous_revision"]
)
return {"rollback_status": result["status"]}
return {"action": "manual_investigation_required"}
The @step decorator breaks the workflow into discrete, observable stages. This is crucial for DevOps—you need to see exactly where in the triage process an incident is, and you need the ability to pause before executing destructive actions like rollbacks.
Step 3: Add a Human-in-the-Loop Gate
This was my first mistake. I initially let the agent auto-execute rollbacks. Within a day, it rolled back a deployment that was actually fine—the alert was caused by a downstream dependency, not the deploy itself. I quickly added a human approval gate:
@step(requires_approval=True)
def execute_rollback(self, deploy_info: dict):
"""Trigger rollback - requires human approval"""
if deploy_info.get("suspect_deploy"):
result = self.tools.argocd_rollback(
app_name=deploy_info["details"]["app_name"],
revision=deploy_info["details"]["previous_revision"]
)
return {"rollback_status": result["status"]}
return {"action": "manual_investigation_required"}
The requires_approval=True flag pauses the workflow and sends a notification to the on-call channel. The engineer can review the context the agent gathered and approve or reject the rollback. This single feature changed the tool from "too risky for production" to "genuinely useful."
Deploying and Running the Agent
With the agent defined, I deployed it using the ADK CLI:
orchestrate agents deploy devops-remediation --env production
The ADK handles packaging the agent, registering it with the watsonx Orchestrate control plane, and wiring up the tool connections. I had to configure authentication for each tool (PagerDuty API key, ServiceNow credentials, etc.) through the platform's credential store—no hardcoding secrets.
Once deployed, the agent can be triggered via the Orchestrate chat interface, a REST API, or by wiring it to listen for PagerDuty webhooks. I chose the webhook route so incidents trigger automatically.
What Actually Happened in Production
After two weeks of running this with human-in-the-loop, here's what I observed:
- Triage time dropped from ~45 minutes to ~3 minutes. The agent gathers all the context (alert details, CMDB info, recent deploys) in seconds. Engineers just review the summary.
- We caught two actual bad deploys early. The agent correctly correlated alerts with recent revisions, and engineers approved the rollbacks.
- False positive rate was about 30%. The agent flagged deploys as suspicious when the real cause was elsewhere. I mitigated this by adding a step that checks downstream service health before flagging a deploy.
Going Further: Multi-Agent Coordination
Where watsonx Orchestrate gets interesting is coordinating multiple agents. After the remediation agent, I built a second agent that handles post-incident review—gathering logs, timelines, and generating a draft incident report in Confluence.
In the orchestration layer, I connected them so that when an incident resolves (either via rollback or manual fix), the review agent automatically kicks off:
workflows:
- name: incident_lifecycle
triggers:
- agent: incident_remediation
event: incident_resolved
steps:
- agent: incident_review
input: "{{ trigger.incident_data }}"
This is where the "control plane" concept clicks. You're not just building isolated agents—you're defining how they hand off work to each other, with observability and governance over the entire chain.
Practical Tips
- Start with read-only agents. Build agents that gather information before you let them take action. The triage agent (steps 1 and 2) was useful on its own, even without the rollback step.
- Invest time in your OpenAPI specs. The quality of your tool integrations depends entirely on how well you describe your APIs. Vague specs lead to confused agents.
- Use the visual builder for non-coders. Our ops team modified workflow logic in the visual builder without needing to touch Python. This was a big win for adoption.
- Always add approval gates for destructive actions. Even if you're confident in your agent's logic, production systems deserve a human checkpoint.
- Test with synthetic incidents first. Don't wait for a real 2 AM alert to find out your agent breaks on edge cases.
Honest Limitations
watsonx Orchestrate isn't a silver bullet. The learning curve for the ADK is steeper than the marketing suggests—the documentation is still maturing, and I spent too much time figuring out the right way to structure multi-step workflows. The platform is also firmly enterprise-priced; if you're a small DevOps team, this might be overkill compared to simpler automation tools. And while it integrates with many SaaS tools out of the box, custom integrations still require writing and maintaining OpenAPI specs, which is tedious.
The biggest conceptual shift is trusting an orchestration layer with your production workflows. It requires rigorous testing, clear approval gates, and a willingness to iterate. But if your DevOps team is drowning in tool sprawl and manual context-switching, watsonx Orchestrate offers a structured way to connect those tools into coherent, observable workflows. For us, cutting incident triage from 45 minutes to 3 minutes made the investment worthwhile.