Monitoring production systems has always relied on agents — but last year the word “agent” split into two meanings. There’s traditional monitoring agent software you install on hosts (CloudWatch agent, Datadog agent, OTel Collector), and there’s the new kind: AI monitoring agents that investigate incidents autonomously.
This post is about deploying the second kind on real production infrastructure — the architecture we ran, how the agent used our runbooks and Jira, what MCP servers we exposed to it through ContextForge, and the things that went wrong before it went right.
Where we started
Our platform team was carrying a familiar problem: around 200 pageable alerts a month across ~40 services on AWS (ECS plus a couple of EKS clusters), and the first half hour of every incident was the same manual loop — check what deployed, query CloudWatch, grep logs, find the runbook, start a Jira ticket, paste screenshots into Slack. Senior engineers were the correlation engine, and it showed in on-call morale.
The collection layer wasn’t the problem. Our monitoring agent software — CloudWatch agent and the OpenTelemetry Collector on the ECS side, Prometheus, Loki, and Grafana on the EKS clusters — already gathered everything we needed. The gap was the jobs between an alert firing and a human understanding it: correlation, diagnosis, documentation. Those are the monitoring agent jobs we automated.
The architecture we deployed
The trigger path is boring on purpose: CloudWatch alarm → SNS topic → Lambda. The Lambda runs an investigation loop against an LLM on Amazon Bedrock with a 5-minute timeout and a hard cap of 8 tool calls per incident. A DynamoDB conditional write dedupes alarm storms — one root cause used to fire 12–15 alarms, and we did not want 15 parallel agents investigating the same outage.
The interesting part is what the agent can touch, and how.
MCP servers via ContextForge
Early prototypes had the Lambda calling boto3 directly. It worked, but every new capability meant new code, new IAM statements, and no clean audit story. We moved to the Model Context Protocol and put ContextForge (IBM’s open-source MCP gateway) in front of everything. The agent sees exactly one endpoint; ContextForge federates the actual servers behind it.
The MCP servers we registered:
| MCP server | What the agent gets | Access level |
|---|---|---|
aws-cloudwatch | GetMetricData, Logs Insights queries | Read-only |
aws-cloudtrail | Recent write events (deploys, config changes) | Read-only |
aws-xray | Trace summaries for latency investigations | Read-only |
aws-ecs / aws-eks | Service, task, and cluster state | Read-only |
kubernetes | K8s API — pod status, events, recent restarts | Read-only |
prometheus | PromQL queries against our EKS metrics | Read-only |
loki | LogQL queries for container logs | Read-only |
grafana | Dashboard and panel lookups for evidence links | Read-only |
confluence-runbooks | Search + fetch runbook pages by failure class | Read-only |
atlassian-jira | Create incident ticket, append comments, set labels | Scoped write |
slack | Post diagnosis to the incident channel | Scoped write |
Our stack is split-brain like most real ones: ECS services report through CloudWatch, while the EKS clusters run the Prometheus/Loki/Grafana stack. The agent doesn’t care — through the gateway it queries PromQL and Logs Insights in the same investigation, which is something none of our dashboards could do for us.
Three ContextForge features earned their keep. Tool allowlisting: we composed a “virtual server” that exposes only the specific tools above, so even though the Atlassian MCP server supports dozens of operations, our agent can literally only create and comment on tickets in one Jira project. Central auth: service-account credentials live in the gateway, not in the Lambda — rotating a token is a gateway config change. Audit logging: every tool call the agent makes is logged in one place, which turned our security review from a three-week argument into a one-hour walkthrough. The read-only IAM on the AWS side plus the gateway allowlist means a prompt injection via a malicious log line has effectively zero blast radius.
How the agent uses runbooks
This was the single biggest accuracy improvement, and I almost skipped it.
Our runbooks lived in Confluence, written for humans. We exposed them through the confluence-runbooks MCP server with a search tool, and changed the agent’s system prompt to make runbook lookup step two of every investigation (step one is always CloudTrail — most incidents are a change someone made).
The flow: alarm fires for checkout-service p99 latency → agent searches runbooks for that service and symptom → finds “Checkout latency — known causes” → the runbook says check RDS connection pool saturation first, it’s bitten us three times → agent runs exactly that Logs Insights query instead of guessing. Diagnoses started citing runbook sections, which did something unexpected for trust: engineers stopped treating the agent as a black box because it was visibly following their own documented procedures.
Two lessons here. First, runbook quality became agent accuracy — services with good runbooks hit ~85% diagnosis accuracy in our grading; services with stale runbooks sat around 60% until we fixed the docs. The agent quietly became a forcing function for runbook hygiene. Second, we added a rule to the prompt: if the runbook and the live evidence disagree, say so explicitly. That caught two genuinely outdated runbooks in the first month.
How the agent updates Jira
We didn’t want another Slack-only tool that leaves no paper trail, so the agent owns the incident ticket lifecycle through the Atlassian MCP server:
- On trigger, it creates a Jira incident ticket: alarm details, affected service, initial severity guess, label
agent-investigated. - During investigation, each significant finding lands as a ticket comment — the CloudTrail event it suspects, the metric window, the log query and result snippet. If a second correlated alarm arrives inside the dedupe window, it’s linked to the same ticket instead of spawning a new one.
- On conclusion, it posts the structured diagnosis — root cause, evidence, suggested action — as the final comment, updates severity based on what it actually found, and drops the ticket link into the Slack post.
The unglamorous payoff: post-incident reviews stopped starting with “let me reconstruct the timeline.” The timeline writes itself, timestamped, while the incident is happening. Our postmortem prep time roughly halved, and the agent-investigated label gave us a free dataset for grading accuracy per failure class.
What went wrong
Honesty section, because vendor posts never include one:
- Confidently wrong diagnoses. In week two, the agent blamed a Redis memory alert on a deploy that turned out to be a red herring; the real cause was a slow key-eviction misconfiguration. This is why we ran shadow mode for a month — agent posts alongside normal alerts, humans grade every diagnosis 👍/👎 in Slack, nothing is suppressed until a failure class grades consistently well.
- Token blowouts. Logs Insights happily returns megabytes. We cap every tool result at 8KB before it enters context; before that cap, one noisy incident cost us $4 in tokens instead of the usual few cents.
- Alert storm concurrency. Before the DynamoDB dedupe, one bad deploy spawned 11 agents investigating in parallel. Reserved Lambda concurrency of 3 plus the dedupe key fixed it.
Results and costs
After three months: pages reaching humans down roughly 70% (correlation plus suppression of graded failure classes), and time-to-first-diagnosis down from 20–30 minutes of human investigation to under 2 minutes of agent runtime. Running cost is ~$120/month — Bedrock tokens plus Lambda plus a negligible DynamoDB table — against the 2–6 engineer-weeks it took to build v1 and maybe 0.15 FTE ongoing for prompt tuning and grading.
For comparison, commercial AIOps platforms doing the correlation half of this commonly run mid-five to six figures annually, licensed per user or per event — meaning you pay more when you’re noisier. What they can’t do at any price is read your runbooks and follow your team’s own diagnostic procedures. That context turned out to be the whole ballgame.
If you’re deploying this
- Get the collection layer right first — your existing monitoring agent software (OTel, CloudWatch agent) is the foundation; the AI agent only reasons over what’s collected.
- Start read-only, one noisy service, shadow mode, grade for a month. Suppress nothing until the grades say you can.
- Put an MCP gateway like ContextForge between the agent and your tools from day one. Allowlisting and central audit logs are what make security reviews short and capability additions cheap.
- Wire runbooks in before you tune anything else — it’s the highest accuracy-per-effort change available.
- Make Jira the system of record, not Slack. Future-you at the postmortem will be grateful.
FAQ
What is a monitoring agent? Either lightweight software installed on hosts to collect metrics, logs, and traces, or an AI agent that investigates alerts and diagnoses root causes on top of that telemetry. Production infrastructure in 2026 typically needs both.
What jobs do monitoring agents perform? Collection, detection, correlation, diagnosis, and documentation/remediation. Traditional monitoring agent software covers the first two; AI monitoring agents automate the last three.
Is it safe to give an AI agent access to production? Deploy read-only via an MCP gateway with tool allowlisting: the agent queries metrics, logs, traces, and change history but its only write scopes are Jira comments and Slack posts. Remediation actions come later, human-approved, per proven failure class.
How much does monitoring using agents cost for production infrastructure? Our deployment runs about $120/month in AWS costs plus initial engineering time. Commercial AIOps platforms range from ~$10K to well over $100K per year.
The interesting part is what the agent can touch, and how.