The Immune System: How to Build AI Systems That Heal Themselves

You are a reliability engineer designing health sentinels
for an AI system. Sentinels are lightweight checks that
run AFTER each pipeline execution and flag anomalies
BEFORE outputs reach consumers.

SYSTEM DESCRIPTION:
[Describe your AI system: inputs, processing steps,
outputs, and who consumes those outputs.]

KNOWN FAILURE MODES:
[Paste your failure map from Issue #27's Resilience
Scorecard, or list the top 5 ways your system breaks.]

Design a sentinel for EACH of these categories:

## 1. FRESHNESS SENTINELS
For each data source and output:
- Max acceptable age (e.g., "signals must be <4 hours old")
- Check method: timestamp comparison, file modification time,
  or API response header
- Alert threshold vs auto-heal threshold
  (e.g., 4hr = warn, 8hr = use fallback data)

## 2. SANITY SENTINELS
For each output your system produces:
- Range checks: "Score must be between 0 and 100"
- Distribution checks: "At least 5 BUY and 5 SELL signals
  expected on a normal day. Zero of either = anomaly."
- Consistency checks: "If VIX > 30 and system says
  STRONG_BUY on 50 tickers, something is wrong."
- Delta checks: "Output changed by >50% from yesterday
  with no corresponding market move = anomaly."

## 3. DEPENDENCY SENTINELS
For each external service:
- Health check endpoint or test query
- Timeout threshold (in seconds, not minutes)
- Fallback behavior when dependency is degraded vs down
- Circuit breaker threshold: "After N failures in M minutes,
  stop trying and use cached data."

## 4. INTEGRITY SENTINELS
For the pipeline itself:
- Input/output count validation: "Processed N inputs,
  produced M outputs. M should be within 10% of N."
- Schema validation: "Output JSON must match this schema."
- Duplicate detection: "Same output produced twice = error."
- Regression detection: "Model accuracy dropped >5%
  from rolling 30-day average."

OUTPUT FORMAT (per sentinel):
- Name: [SHORT_NAME] (e.g., FRESH_SIGNALS)
- Layer: Freshness / Sanity / Dependency / Integrity
- Check: What it verifies (one sentence)
- Frequency: After every run / hourly / daily
- Alert level: INFO / WARN / CRITICAL
- Auto-heal action: What to do if triggered (or "none — page human")
- Implementation: Pseudocode (10 lines max)

You are building an automated diagnosis engine for an AI
system. The engine takes sentinel alerts as input and
produces a specific, actionable root cause diagnosis.

SENTINEL ALERTS:
[Paste current alerts, or describe a scenario:
"FRESH_SIGNALS fired (data 6hrs old), SANITY_SCORE_RANGE
fired (3 scores outside bounds), DEPENDENCY_API healthy"]

KNOWN FAILURE PATTERNS:
Build a pattern library — a decision tree that maps
combinations of sentinel alerts to root causes:

## PATTERN MATCHING RULES

For each known failure mode, define:

Pattern: [NAME]
Trigger: [Which sentinels fire, in what combination]
Root cause: [One sentence — the actual problem]
Confidence: HIGH / MEDIUM / LOW
Evidence needed: [What additional data to collect to
  confirm the diagnosis]
Disambiguation: [How to tell this apart from similar
  patterns]

Example patterns to define:

1. STALE_SOURCE
   Trigger: FRESH_* fires + DEPENDENCY_* healthy
   Root cause: Upstream source is online but publishing
   stale data. API returns 200 but content is old.
   Confidence: HIGH if source timestamp confirms staleness.

2. PROVIDER_OUTAGE
   Trigger: DEPENDENCY_* fires + FRESH_* fires shortly after
   Root cause: External service down, causing cascade.
   Confidence: HIGH if dependency health check fails.

3. CONFIG_DRIFT
   Trigger: SANITY_* fires + FRESH_* healthy + DEPENDENCY_*
   healthy (all inputs fresh and sources up, but output
   is wrong)
   Root cause: A parameter or threshold was changed.
   Confidence: MEDIUM — needs config diff to confirm.

4. DATA_CORRUPTION
   Trigger: INTEGRITY_* fires (schema validation, count
   mismatch, duplicates)
   Root cause: Input data is structurally broken.
   Confidence: HIGH if schema validation pinpoints field.

5. CAPACITY_OVERLOAD
   Trigger: Pipeline duration > 2x normal + INTEGRITY
   count mismatch (some items dropped)
   Root cause: Volume exceeded processing capacity.
   Confidence: HIGH if input count confirms spike.

6. FEEDBACK_LOOP_DRIFT
   Trigger: REGRESSION_* fires (accuracy declining) +
   all other sentinels healthy
   Root cause: The self-improvement loop is learning
   wrong lessons from its own outputs.
   Confidence: LOW — hardest to diagnose. Requires
   manual review of recent learning updates.

## UNKNOWN PATTERN HANDLER
When sentinel alerts do not match any known pattern:
- Log all active alerts + system state snapshot
- Classify as UNKNOWN with confidence LOW
- Page human with full context
- After resolution: add the new pattern to the library

OUTPUT: For each diagnosed pattern, produce:
{
  "pattern": "STALE_SOURCE",
  "confidence": "HIGH",
  "root_cause": "...",
  "evidence": ["..."],
  "recommended_response": "...",
  "escalate_to_human": true/false
}

You are a site reliability engineer designing automated
response playbooks for an AI system. Each playbook
handles one diagnosed failure pattern end-to-end.

DIAGNOSIS: [Paste output from the Diagnosis Engine]

For each failure pattern, design a response playbook:

## PLAYBOOK STRUCTURE

### Immediate Response (seconds)
What to do RIGHT NOW to stop the bleeding:
- Halt: Should the pipeline stop processing? (Y/N + why)
- Quarantine: Should current outputs be quarantined
  (marked as potentially bad) before reaching consumers?
- Notify: Who/what gets notified? (Slack, email, dashboard)

### Automated Fix (minutes)
Pre-scripted remediation steps:

For STALE_SOURCE:
  1. Switch to backup data source (if available)
  2. If no backup: use last-known-good cached data
  3. Tag all outputs as "DEGRADED — stale source"
  4. Set retry timer to re-check primary source every 5min
  5. After 3 successful checks: restore to primary

For PROVIDER_OUTAGE:
  1. Activate circuit breaker (stop retries)
  2. Switch to fallback mode (cached responses, reduced
     feature set, or graceful degradation)
  3. Monitor provider status endpoint every 60s
  4. After 3 consecutive successes: close circuit breaker
  5. Run integrity check on first post-recovery output

For CONFIG_DRIFT:
  1. Snapshot current config
  2. Diff against last-known-good config
  3. If diff is small (<3 parameters): auto-revert
  4. If diff is large: quarantine outputs + page human
  5. Log the drift event for pattern analysis

For DATA_CORRUPTION:
  1. Reject corrupted input batch
  2. Attempt re-fetch from source
  3. If re-fetch also corrupt: use last-known-good data
  4. Flag affected outputs for manual review
  5. Add corruption signature to integrity sentinel

For CAPACITY_OVERLOAD:
  1. Enable rate limiting on input queue
  2. Process in priority order (highest value first)
  3. Drop or defer lowest-priority items
  4. Alert if >20% of items are dropped
  5. Post-incident: adjust capacity thresholds

For FEEDBACK_LOOP_DRIFT:
  1. FREEZE the learning loop (stop all parameter updates)
  2. Snapshot current model state
  3. Compare outputs to 7-day-ago model state
  4. Page human with divergence report
  5. Do NOT auto-fix — this requires human judgment

### Recovery Verification (minutes to hours)
After the fix is applied:
- Re-run sentinels to verify fix worked
- Compare output quality to pre-incident baseline
- Confirm no downstream consumers received bad data
- Update incident log with timeline + resolution

### Post-Incident Learning
After every incident (automated or manual):
- Was the diagnosis correct?
- Was the response appropriate?
- How long from detection to resolution?
- Should this pattern's response be upgraded?
- Add any new sentinel patterns discovered

OUTPUT per playbook:
- Pattern name
- Response time target (how fast should this resolve?)
- Automation level: FULL (no human needed) / PARTIAL
  (human approves before fix executes) / MANUAL (human
  required)
- Rollback plan: How to undo the automated response
  if it makes things worse
- Success criteria: How to know the fix worked