The Tool Surface Problem

An SRE AI agent is only as useful as the data it can reach. In 2025, building agent tool integrations meant writing custom API wrappers for every observability backend — Elasticsearch queries via REST, Prometheus via PromQL HTTP API, Kubernetes via kubectl subprocesses. Each integration was bespoke, brittle, and unmaintainable.

Model Context Protocol (MCP) changes this. MCP is an open standard, introduced by Anthropic in late 2024, that defines a universal interface between AI agents and external tools. An observability backend that exposes an MCP server can be connected to any MCP-compatible agent in minutes — no custom wrapper, no API translation layer.

By mid-2026, the major observability vendors have shipped MCP servers: Elastic, Honeycomb, Grafana, Datadog, and the Kubernetes community have all published official or community MCP servers. This dramatically lowers the barrier to building a well-instrumented SRE agent.

Core MCP Tools for a Typical SRE Stack

A production SRE agent in 2026 typically has access to four categories of MCP tools. Each one expands the agent's ability to form and verify hypotheses during an incident investigation.

Logs — Elastic MCP

Full-text and structured search

Natural language queries converted to KQL or Lucene. Time-windowed log retrieval, error rate aggregation, pattern clustering. Agent can ask "show me all 5xx errors from auth-service in the 10 minutes before the alert" without knowing the index schema.

Metrics — Prometheus / Grafana MCP

Time-series queries and anomaly detection

PromQL generation from natural language, metric exploration, threshold comparison. Agent can identify when a metric crossed a threshold relative to a deployment timestamp without having PromQL expertise.

Traces — Honeycomb MCP

Request-level latency and error tracing

Distributed trace retrieval, span-level error analysis, latency heatmaps. Honeycomb's column-oriented trace storage makes arbitrary slicing fast — the agent can find the exact span where a request started failing.

Infrastructure — Kubernetes MCP

Pod state, events, and resource usage

Read-only access to pod logs, events, resource limits, node pressure, and recent rollout history. Agent can identify OOMKilled pods, crashloop backoffs, or pending pods without needing kubectl access or cluster context files.

What an MCP-Enabled Investigation Looks Like

Here is a representative trace of how an MCP-equipped SRE agent investigates a latency spike. Each step corresponds to a distinct MCP tool call, and the agent reasons between steps to decide what to query next.

  1. Alert received: p99 latency for checkout-service exceeded 2s. Agent receives alert context: service name, metric value, timestamp, linked runbook.
  2. Prometheus MCP: Query latency histogram for checkout-service over the past 30 min. Find: p99 crossed threshold at 14:22 UTC, coinciding with a spike in db_query_duration.
  3. Kubernetes MCP: List recent rollouts and pod events for checkout-service. Find: no recent deployment. Find: two pods showing high memory pressure on node pool az-eastus-2.
  4. Elastic MCP: Search logs for checkout-service between 14:20–14:25. Find: 847 occurrences of "slow query: 1200ms" from a specific SQL statement hash.
  5. Honeycomb MCP: Retrieve traces for the slow SQL hash. Find: query is a sequential scan on the orders table. EXPLAIN output in trace attributes shows a missing index after a schema migration 48 hours ago.
  6. Synthesis: Agent forms hypothesis: schema migration dropped an index. Queries deployment history to confirm migration timestamp. Produces structured output with confidence 0.91. Proposed action: recreate index (requires DBA approval).

Without MCP, each of those tool calls would have required the agent to know API endpoints, authentication schemes, query languages, and response formats for five different systems. With MCP, the agent describes what it needs, and the MCP server handles the translation.

Wiring MCP into an Agent: The Configuration

The following shows how MCP servers are declared in a Claude agent configuration. The agent runtime handles tool discovery, schema loading, and request routing automatically once the servers are declared.

# agent-oncall.yaml — MCP server declarations name: oncall-triage-agent model: claude-opus-4-7 mcp_servers: - name: elastic transport: http url: https://mcp.internal/elastic auth: bearer scopes: [logs:read] - name: prometheus transport: http url: https://mcp.internal/prometheus auth: bearer scopes: [metrics:read] - name: honeycomb transport: http url: https://mcp.internal/honeycomb auth: bearer scopes: [traces:read] - name: kubernetes transport: http url: https://mcp.internal/k8s auth: bearer scopes: [pods:read, events:read] # no write access guardrails: no_write_actions: true max_tool_calls: 30 timeout_seconds: 120

Pre-Deployment Health Checks via MCP

One underused application of MCP in SRE workflows is pre-deployment health verification. Before a deployment proceeds, an agent queries the observability stack to confirm baseline health: error rates within tolerance, no active alerts, resource headroom on target nodes, and no open incidents linked to the deploying service.

This is particularly valuable for off-hours deployments and automated release pipelines where no engineer is actively watching. The agent's pre-deployment report becomes a signed artifact attached to the deployment record — evidence that the system was in a known state when the deployment started.

Pre-Deployment Check Pattern

1. Query error rate baseline for target service (last 24h p99). 2. Check for active alerts in any upstream dependency. 3. Verify node resource headroom exceeds 20%. 4. Confirm no open incidents tagged with the service. 5. Generate signed health report. If any check fails, block deployment and alert on-call lead — do not proceed autonomously.

Security Boundaries for MCP Tool Access

Giving an AI agent broad observability access creates a real risk surface. A compromised agent — whether through prompt injection via a log entry, a malicious trace attribute, or a poisoned runbook — could exfiltrate sensitive log data, enumerate internal service topology, or be used to confirm whether a breach has been detected.

MCP Security Boundaries

Scope MCP tokens to read-only. There is no reason for a triage agent to have write access to your observability backend. Most production log data is sensitive — MCP tokens should be scoped to the minimum required query surface.

Treat log data as untrusted agent input. A log entry is user-controlled data. If your service logs request bodies or user-provided strings, an attacker can craft a log message containing prompt injection. Sanitize what the agent receives; log retrieval tools should strip or escape instruction-like patterns.

Audit agent tool calls. Every MCP tool call should be logged with the agent session ID, the query issued, and the result size. This creates an audit trail for investigating agent behavior during or after a security incident.

The Payoff

Teams that have wired full MCP observability stacks into their SRE agents report that the most significant change is not speed — it is the quality of hypotheses. When an agent can query logs, metrics, traces, and infrastructure state in a single investigation thread, it surfaces correlations that would take a human engineer 20-30 minutes to find manually. The agent does not replace the engineer's judgment; it replaces the 3am data gathering that precedes that judgment.

The practical advice for teams starting this journey: wire one backend at a time, starting with logs. Get the agent producing useful log summaries before adding metrics. Add traces once the log and metric investigation is reliable. Add Kubernetes last — it has the highest blast potential from a security perspective and benefits most from a mature guardrail configuration.