AI-Driven SRE & Chaos Tracing: Designing an Autonomous Failure Lab with OpenTelemetry

By Michael Mifsud

Architecting for Failure in Legacy Environments

In modern distributed systems, failure is an inevitability rather than an anomaly. Mismatched microservices, network latency, resource limits, database connection pool exhaustion, and third-party API outrages can combine to trigger catastrophic cascading failures. Testing for these scenarios in enterprise-grade platforms—especially legacy content engines like Alfresco, which integrate multiple stateful dependencies like Solr search indexers, ActiveMQ message brokers, and transactional relational databases—is historically difficult.

To confidently manage these stacks, organizations are moving from reactive disaster recovery to active Chaos Engineering and AI-Driven Site Reliability Engineering (SRE).

The AI-Driven SRE & Chaos Engineering Lab is a sandboxed testing framework designed to simulate high-stress failure modes in an Alfresco cluster. By combining active fault injection, high-resolution OpenTelemetry (OTel) distributed tracing, Prometheus-backed metric monitoring, and an LLM-powered AI Advisor, the platform can actively trigger system anomalies and automatically diagnose the root cause in real time.

The Chaos Lab Architecture

The lab is orchestrated as a modular, version-locked environment, clearly separating the chaos control plane, the target system under test, the telemetry monitoring cluster, and the AI diagnostics agent.

AI SRE Advisor

Observability Suite

Alfresco Stack (SUT)

Chaos Injection & Testing

Control Plane & Chaos UI

Metrics

Postgres Exporter

Traces (OTLP/HTTP)

MuleSoft API Burst

Postgres Hammer

Chaos Dashboard (Port 8087)

Chaos API (Port 8086)

Chaos Runner (Python + OTel)

Nginx Proxy (Port 8082)

Alfresco Repo (Port 8080)

PostgreSQL (Port 5432)

Solr Search (Port 8983)

ActiveMQ (Port 61616)

Prometheus (Port 9091)

Grafana Tempo (Port 4318)

Grafana UI (Port 3001)

SRE Advisor (Anthropic Claude)

1. Trace-Integrated Chaos: OpenTelemetry & Tempo

A common problem in traditional chaos experiments is “blind injection.” When load tests run, the SRE team can see CPU or latency graphs spike in Prometheus, but they struggle to correlate specific anomalies with a specific chaos event injection.

The Chaos Lab solves this by wrapping stress injectors in OpenTelemetry distributed tracing. The Python-based ChaosRunner uses the OpenTelemetry Python SDK to publish span metrics directly to Grafana Tempo (listening on OTLP/HTTP port 4318).

Recording Chaos Attributes in distributed Traces

When a load experiment begins, the runner establishes a root span (sync_and_execute). Inside this span, sub-tasks like API burst hammering or SQL thread pool hammering are tracked as child spans. Crucially, the runner injects the exact variables—such as concurrency thresholds and stress states—directly into the trace metadata:

import time
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

# 1. Identify service inside OTel context
resource = Resource(attributes={"service.name": "chaos-engine"})
provider = TracerProvider(resource=resource)

# 2. Bind the OTLP/HTTP Exporter to Grafana Tempo
exporter = OTLPSpanExporter(endpoint="http://localhost:4318/v1/traces")
provider.add_span_processor(BatchSpanProcessor(exporter))
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)

Inside the execution loop, the root span and children record the load attributes:

def trigger_mule_burst(self, concurrency):
	with tracer.start_as_current_span("mule_burst") as span:
		span.set_attribute("chaos.mule_concurrency", concurrency)
		
		def attack():
			try:
				requests.post(self.cfg.MULE_URL, timeout=2)
			except:
				pass
		
		with ThreadPoolExecutor(max_workers=max(1, concurrency)) as executor:
			for _ in range(concurrency):
				executor.submit(attack)

def sync_and_execute(self):
	with tracer.start_as_current_span("sync_and_execute") as span:
		try:
			r = requests.get(self.cfg.STATE_API_URL + "/state", timeout=2)
			state = r.json()
			
			concurrency = int(state.get('mule_concurrency', 1))
			self.db_stressing = state.get('db_stress', False)
			
			# Enforce contextual traceability in Grafana Tempo
			span.set_attribute("chaos.concurrency_target", concurrency)
			span.set_attribute("chaos.db_stress_enabled", self.db_stressing)
			
			self.trigger_mule_burst(concurrency)
			if self.db_stressing:
				self.hammer_db()
				
		except Exception as e:
			# Record runtime errors directly to the trace
			span.record_exception(e)
			span.set_status(trace.Status(trace.StatusCode.ERROR))

In Grafana, this telemetry correlation allows operators to look at a slow SQL query or an HTTP 504 error and immediately view the active OTel spans to see if a DB Hammering experiment (chaos.db_stress_enabled = true) was actively running at that exact millisecond.

2. LLM-Powered Automated Root-Cause Analysis

Having metrics and traces is valuable, but diagnosing failures during high-severity outrages is a race against time. The Chaos Lab introduces an AI SRE Advisor that leverages Anthropic’s Claude API to parse system states and compile instantaneous root-cause analysis (RCA).

The SRE Advisor script acts as an autonomous diagnostic agent. It communicates with the Chaos Control API to fetch active variables, gathers live performance symptoms, and structures this data into a highly contextual SRE prompt.

import os
import anthropic
import requests

class SREAdvisor:
    def __init__(self, api_key=None):
        self.key = api_key or os.getenv("ANTHROPIC_API_KEY")
        self.client = anthropic.Anthropic(api_key=self.key) if self.key else None

    def analyze_chaos(self, concurrency, db_stress, symptoms):
        if not self.client:
            return "AI SRE Error: Anthropic API Key missing."

        prompt = f"""
        Senior SRE Analysis Request:
        System Load: {concurrency} threads.
        DB Stress Worker: {'Active' if db_stress else 'Inactive'}.
        Symptoms: {symptoms}.

        Provide a 2-sentence RCA and one immediate mitigation step.
        """

        message = self.client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=150,
            messages=[{"role": "user", "content": prompt}]
        )

        return message.content[0].text

The script can be executed as a CLI tool or scheduled cron loop:

# When executed, the advisor queries the API and reports:
# 🔍 Fetching live system state from Chaos API...
# 🤖 AI SRE Advisor is analyzing (Current Load: 45 threads)...
# ------------------------------
# RCA: The concurrent MuleSoft integration tests (45 threads) are causing Tomcat thread pool starvation on the primary Alfresco repository, while the background database hammer is exhausting connection limits.
# MITIGATION: Enable active circuit-breakers on the MuleSoft API client and scale Tomcat maxThreads from 50 to 200.
# ------------------------------

This represents a key architectural shift for SRE: moving from human-curated dashboards to closed-loop autonomous diagnostic agents capable of reading telemetry and recommending mitigations.

3. Reliability and IaC Standards

To ensure the chaos sandboxing platform is repeatable, safe, and robust, the infrastructure complies with strict cloud-native reliability standards: