∇GradientSchool
Intermediate
15 min read
#generative ai#Guide

AI Reliability & SRE

Comprehensive guide on AI Reliability & SRE.

AI Reliability & SRE

1. Learning Objectives#

By the end of this notebook, you should be able to:

  1. Explain why Generative AI systems require specialized reliability engineering.
  2. Distinguish availability, reliability, quality, safety, and correctness.
  3. Design AI-specific Service Level Objectives (SLOs).
  4. Identify common failure modes across models, RAG, agents, tools, and infrastructure.
  5. Design timeouts, retries, backoff, circuit breakers, and fallbacks.
  6. Understand latency components such as queue time, prefill, decode, retrieval, and tool execution.
  7. Design reliable AI gateways and model-provider abstractions.
  8. Build resilient RAG systems.
  9. Design reliable agent workflows with bounded execution.
  10. Apply queueing, admission control, backpressure, and load shedding.
  11. Perform capacity planning and load testing for AI workloads.
  12. Design observability using logs, metrics, traces, and AI quality signals.
  13. Handle model, provider, GPU, database, and dependency outages.
  14. Design disaster recovery and business continuity for AI systems.
  15. Build incident-response and post-incident learning processes.
  16. Design production-grade AI reliability architectures.

2. What Is AI Reliability?

Reliability is the ability of a system to consistently provide an acceptable result when required.

For a traditional API:

Architecture & Data Flow
Request
 |
 v
Service
 |
 v
Response

For an AI system:

Architecture & Data Flow
Request
 |
 v
Gateway
 |
 +--> Model
 +--> Retrieval
 +--> Tools
 +--> Databases
 +--> External APIs
 |
 v
Generation
 |
 v
Validation
 |
 v
Response

There are many more ways to fail.

3. Reliability Is More Than Uptime

A system can be available but unreliable.

Example:

text
HTTP 200
+
Wrong answer

Technically:

Mathematical Formulation
Available = Yes

Operationally:

Mathematical Formulation
Useful = No

AI reliability therefore includes:

  • availability
  • latency
  • correctness
  • groundedness
  • safety
  • consistency
  • tool success
  • data freshness

4. AI Reliability Model

A useful mental model:

Architecture & Data Flow
Reliability
 |
 +--> Infrastructure reliability
 |
 +--> Model reliability
 |
 +--> Data reliability
 |
 +--> Retrieval reliability
 |
 +--> Tool reliability
 |
 +--> Workflow reliability
 |
 +--> Safety reliability
 |
 +--> User experience

The weakest component can dominate the complete workflow.

5. Reliability vs Availability

Availability#

Was the service reachable?

Architecture & Data Flow
HTTP request
 |
 v
HTTP response

Reliability#

Did the system perform the intended task correctly?

Architecture & Data Flow
Request
 |
 v
Successful execution
 |
 v
Correct / useful outcome

For AI systems, both matter.

6. Reliability vs Correctness

Suppose an assistant responds every time:

›100% response rate

But:

›20% answers are incorrect

The service is highly available but not sufficiently reliable for a factual enterprise workflow.

Therefore:

Mathematical Formulation
Technical availability
+
AI quality
=
Meaningful service reliability

7. Reliability vs Safety

A system can generate correct answers most of the time and still be unsafe.

Examples:

  • unauthorized data exposure
  • dangerous tool execution
  • prompt injection
  • inappropriate automated actions

Safety is therefore a separate reliability dimension.

8. Service Level Indicators

SLIs are measurable signals.

Examples:

text
Availability
Latency
Error rate
Timeout rate
Token throughput
Tool success rate
Retrieval success
Groundedness
Task success
Safety violation rate

9. Service Level Objectives

An SLO defines the target.

Example:

Mathematical Formulation
Availability >= 99.9%
P95 latency <= 2 seconds
Task success >= 95%
Critical safety violations = 0

The SLO should represent user or business expectations.

10. AI SLOs

A mature AI application may define:

Mathematical Formulation
Availability SLO
99.9%

Latency SLO
P95 <= 3 seconds

Groundedness SLO
>= 95%

Tool success SLO
>= 99%

Critical safety incidents
0

Not every application needs every metric.

Choose metrics based on the workflow.

11. Error Budgets

If an availability SLO is:

›99.9%

the permitted unavailability is approximately:

›0.1%

That allowance is the error budget.

Conceptually:

Architecture & Data Flow
SLO
 |
 v
Allowed failure
 |
 v
Error budget

Error budgets help balance reliability work and feature velocity.

12. AI Error Budgets

Error budgets can include:

  • unavailable requests
  • excessive latency
  • failed tool calls
  • incorrect task outcomes
  • groundedness failures

For high-risk systems, safety failures may have effectively zero tolerance.

13. Reliability Targets by Use Case

Use caseImportant reliability dimensions
Chat assistantAvailability, latency
Enterprise RAGGroundedness, retrieval, latency
Customer supportTask success, availability, escalation
Analytics agentSQL correctness, authorization
Coding agentTest success, repository safety
Financial workflowCorrectness, authorization, audit
Autonomous agentTool success, safety, bounded execution

14. AI Request Lifecycle

A typical request:

Architecture & Data Flow
Client
 |
 v
API Gateway
 |
 v
Authentication
 |
 v
Authorization
 |
 v
AI Gateway
 |
 v
Routing
 |
 v
Retrieval
 |
 v
Model
 |
 v
Tool execution
 |
 v
Validation
 |
 v
Response

Each stage should have:

  • timeout
  • error handling
  • observability
  • ownership

15. Failure Domains

A failure domain is a component or boundary where failures can occur.

Examples:

text
Client
Gateway
Model provider
GPU
Vector database
SQL database
Tool API
Network
Queue
Storage
Identity provider

Map these explicitly.

16. Failure Mode Table

ComponentFailureImpact
Model providerOutageGeneration unavailable
Vector DBTimeoutRAG unavailable
Tool APIErrorWorkflow incomplete
GPUFailureCapacity reduction
NetworkPacket lossLatency/errors
IdentityOutageAuthentication failure
QueueSaturationIncreased latency
StorageFailureData unavailable

17. Dependency Mapping

Create a dependency graph:

Architecture & Data Flow
AI Application
 |
 +--> Identity
 |
 +--> AI Gateway
 |
 +--> Model Provider
 |
 +--> Vector DB
 |
 +--> Tool Gateway
 |
 +--> CRM
 +--> ERP

This reveals where reliability risks originate.

18. Timeouts

Every external dependency should have a timeout.

Bad:

🐍 Python
response = model.generate(prompt)

Better conceptually:

🐍 Python
response = model.generate(
 prompt,
 timeout=10,
)

A timeout prevents one dependency from consuming resources indefinitely.

19. Timeout Budgets

Suppose the user-facing target is:

›3 seconds

Do not allocate:

Mathematical Formulation
Model = 3 sec
Retrieval = 2 sec
Tools = 4 sec

The total could exceed the user budget.

Instead:

Mathematical Formulation
Request budget = 3 sec

Gateway 100 ms
Retrieval 400 ms
Model 1800 ms
Validation 200 ms
Network 300 ms

Leave margin for variance.

20. Retries

Retries can recover transient failures.

Examples:

  • temporary network failure
  • rate limit
  • service overload
  • transient database error

But retries can also amplify outages.

Architecture & Data Flow
Service failing
 |
 v
100 clients retry
 |
 v
More load
 |
 v
Service fails harder

This is a retry storm.

21. Exponential Backoff

Instead of:

text
retry immediately
retry immediately
retry immediately

use increasing delays:

text
100 ms
200 ms
400 ms
800 ms

Add jitter so clients do not retry simultaneously.

22. Retry Policy

Retry only when appropriate.

Good retry candidates:

text
Timeout
Temporary network failure
Rate limit
Transient server error

Avoid blindly retrying:

text
Invalid request
Authorization failure
Permanent validation error
Unsafe tool action

23. Idempotency

Retries are dangerous for write operations.

Example:

›Create payment

If the request succeeds but the response is lost, retrying may create a second payment.

Use idempotency keys:

🐍 Python
request = {
 "idempotency_key": "unique-operation-id",
 "action": "create_payment",
}

The downstream system should recognize repeated attempts.

24. Circuit Breakers

A circuit breaker prevents repeated calls to a failing dependency.

Architecture & Data Flow
Normal
 |
 v
Closed
 |
 +--> failures
 |
 v
 Open
 |
 v
 Stop calls
 |
 v
 Half-open
 |
 +--> Healthy --> Closed
 |
 +--> Failed --> Open

This protects both systems.

25. Fallbacks

Possible fallbacks:

Architecture & Data Flow
Primary model
 |
 +--> unavailable
 |
 v
Secondary model
 |
 +--> unavailable
 |
 v
 Cached answer
 |
 v
 Human escalation

Fallbacks should preserve safety and authorization.

26. Model Provider Fallback

An enterprise gateway can support:

text
Provider A
Provider B
Private Model
Local Model

Routing:

Architecture & Data Flow
Primary
 |
 +--> Healthy --> Use
 |
 +--> Failed --> Fallback

Do not automatically send sensitive data to a provider that is not approved for that data class.

27. Graceful Degradation

When a component fails, reduce functionality intentionally.

Example:

Architecture & Data Flow
Full multimodal assistant
 |
 v
Text-only assistant
 |
 v
Search-only mode
 |
 v
Static help / escalation

A degraded but safe experience can be better than complete failure.

28. Load Shedding

When demand exceeds capacity:

Architecture & Data Flow
Traffic
 |
 v
Admission control
 |
 +--> Critical --> Process
 |
 +--> Standard --> Queue
 |
 +--> Low priority --> Delay / reject

Load shedding prevents total system collapse.

29. Backpressure

Backpressure occurs when downstream systems cannot keep up.

Architecture & Data Flow
Producer
 |
 v
Queue
 |
 v
Worker
 |
 v
Slow dependency

Without control:

Architecture & Data Flow
Queue grows
 |
 v
Memory grows
 |
 v
System becomes unstable

Use bounded queues and admission control.

30. Queue-Based Architecture

For asynchronous workloads:

Architecture & Data Flow
Client
 |
 v
API
 |
 v
Queue
 |
 +--> Worker 1
 +--> Worker 2
 +--> Worker 3
 |
 v
Result store

Useful for:

  • document processing
  • video analysis
  • batch inference
  • large evaluations
  • synthetic data generation

31. Synchronous vs Asynchronous

Synchronous#

Architecture & Data Flow
Request
 |
 v
Process
 |
 v
Response

Good for:

  • chat
  • short answers
  • interactive actions

Asynchronous#

Architecture & Data Flow
Request
 |
 v
Job
 |
 v
Queue
 |
 v
Worker
 |
 v
Result

Good for:

  • long-running workflows
  • media processing
  • large document jobs

32. Latency

AI latency can be decomposed:

Mathematical Formulation
Total latency
=
queue
+
network
+
retrieval
+
prefill
+
decode
+
tools
+
validation

Understanding the components makes optimization much easier.

33. Time to First Token

TTFT measures how long until the first generated token appears.

Architecture & Data Flow
Request
 |
 |---- TTFT ----|
 first token

TTFT matters strongly for interactive applications.

34. Time Per Output Token

After the first token, generation continues.

text
Token 1
 |
 Token 2
 |
 Token 3
 |
 ...

Decode speed influences the total response time.

35. Total Latency

A simplified model:

Mathematical Formulation
Total latency
≈
TTFT
+
generation time

For RAG and agents:

Mathematical Formulation
Total latency
=
retrieval
+
model calls
+
tool calls
+
generation

Parallelization can reduce critical-path latency.

36. Parallel Tool Calls

Instead of:

Architecture & Data Flow
Tool A
 |
 v
Tool B
 |
 v
Tool C

when independent:

Architecture & Data Flow
 +--> Tool A --+
 | |
Request+--> Tool B --+--> Continue
 | |
 +--> Tool C --+

This can significantly reduce workflow latency.

37. Critical Path

For a workflow:

›A -> B -> C

latency is approximately:

›A + B + C

For:

Architecture & Data Flow
A
 \
 -> C
 /
B

latency is approximately:

›max(A, B) + C

This is why independent operations should often run in parallel.

38. AI Observability

Three classic pillars:

text
Logs
Metrics
Traces

AI adds:

text
Quality signals
Cost signals
Safety signals

39. AI Logs

Useful structured fields:

🐍 Python
log = {
 "request_id": "req-123",
 "tenant_id": "tenant-1",
 "model": "model-x",
 "latency_ms": 1240,
 "input_tokens": 2100,
 "output_tokens": 350,
 "status": "success",
}

Sensitive content should be redacted or excluded according to policy.

40. Metrics

Track:

text
Request count
Error rate
Timeout rate
P50 latency
P95 latency
P99 latency
Tokens/sec
Queue depth
GPU utilization
Tool success rate
Retrieval latency
Task success
Cost

41. Traces

A distributed trace can show:

Architecture & Data Flow
Request
 |
 +--> Auth
 |
 +--> Retrieval
 | |
 | +--> Vector DB
 |
 +--> Model
 |
 +--> Tool
 |
 +--> CRM

This makes bottlenecks and failure propagation visible.

42. AI Quality Monitoring

Production systems should sample outputs for:

  • groundedness
  • factuality
  • relevance
  • citation correctness
  • task success
  • safety

Not every metric needs to run on every request.

43. Reliability Dashboard

A useful dashboard:

Architecture & Data Flow
+-----------------------------------------------+
| AI Reliability |
+-----------------------------------------------+
| Availability 99.95% |
| P95 latency 1.8 sec |
| Error rate 0.12% |
| Task success 96.2% |
| Groundedness 97.1% |
| Tool success 99.4% |
| Queue depth 24 |
| GPU utilization 71% |
+-----------------------------------------------+

44. Golden Signals

Traditional SRE often uses:

text
Latency
Traffic
Errors
Saturation

For AI add:

text
Quality
Cost
Safety

So an AI reliability dashboard can track:

text
Latency
Traffic
Errors
Saturation
Quality
Cost
Safety

45. Saturation

Saturation indicates how close a resource is to capacity.

Examples:

  • GPU utilization
  • queue depth
  • database connections
  • memory
  • concurrency

High saturation often predicts increased latency and failures.

46. Capacity Planning

Estimate:

text
Expected traffic
Peak traffic
Average input tokens
Average output tokens
Concurrency
Model throughput
GPU capacity
Storage
Database capacity

Then test the design under realistic peaks.

47. Headroom

Do not operate continuously at 100% capacity.

Example:

Mathematical Formulation
Maximum capacity = 100%
Normal target = 60–70%

Headroom provides room for:

  • traffic spikes
  • failures
  • maintenance
  • retries
  • uneven workloads

48. Load Testing

A realistic AI load test should vary:

  • prompt length
  • output length
  • concurrency
  • model
  • retrieval size
  • tool usage
  • multimodal input
  • failure rates

Do not test only one short prompt.

49. Load Test Example

🐍 Python
import time

def run_request(client, prompt):
 start = time.perf_counter()

 response = client.generate(prompt)

 latency = time.perf_counter() - start
 return response, latency

A production load test should execute many concurrent requests and collect distributions rather than a single latency value.

50. Latency Percentiles

Average latency can hide bad experiences.

Example:

Mathematical Formulation
P50 = 1.0 sec
P95 = 3.0 sec
P99 = 10.0 sec

Most users may be fine, while a meaningful tail experiences very slow responses.

Use:

  • P50
  • P95
  • P99

for production analysis.

51. Tail Latency

AI systems often have high variance because of:

  • long prompts
  • long outputs
  • queueing
  • tool calls
  • provider throttling
  • retrieval delays

Tail latency matters especially for interactive systems.

52. Provider Rate Limits

External providers may enforce:

text
Requests/minute
Tokens/minute
Concurrency

When limits are reached:

Architecture & Data Flow
Provider
 |
 v
Rate limit
 |
 v
429

Your gateway should handle this intentionally.

53. Provider Health Monitoring

Track per provider:

text
Availability
Latency
Error rate
Rate limits
Cost
Quality

Then routing can use health information.

text
Provider A
Healthy

Provider B
Degraded

Provider C
Unavailable

54. Model Health

A model can be technically available but behaviorally degraded.

Examples:

  • unusual latency
  • malformed structured output
  • increased refusal rate
  • increased hallucination
  • tool-call failures

Monitor behavior, not only HTTP status.

55. Structured Output Reliability

If an application expects JSON:

json
{
 "priority": "high",
 "category": "billing"
}

validate it.

🐍 Python
from pydantic import BaseModel

class Ticket(BaseModel):
 priority: str
 category: str

Treat model output as untrusted input.

56. Output Validation

A robust pipeline:

Architecture & Data Flow
Model
 |
 v
Schema validation
 |
 +--> Valid --> Continue
 |
 +--> Invalid --> Repair / retry / fallback

Never assume generated output is valid simply because the model was instructed to produce it.

57. Tool Reliability

Tool failures can be caused by:

  • network errors
  • authentication
  • invalid parameters
  • rate limits
  • downstream outages
  • business-rule failures

Every tool should have:

  • timeout
  • schema validation
  • error classification
  • retry policy
  • authorization
  • audit

58. Tool Result Validation

Architecture & Data Flow
Tool
 |
 v
Raw result
 |
 v
Schema validation
 |
 v
Business validation
 |
 v
Agent

The agent should not blindly trust tool output.

59. RAG Reliability

RAG can fail through:

text
Bad ingestion
Bad chunking
Wrong index
Stale data
Permission error
Poor retrieval
Missing evidence
Wrong context

Therefore monitor the full retrieval pipeline.

60. RAG Reliability Signals

Useful metrics:

text
Retrieval latency
Recall@K
Precision@K
MRR
Groundedness
Citation correctness
No-evidence rate
Stale-document rate
Permission-denied retrieval rate

61. No-Evidence Handling

A reliable RAG system should be able to say:

›No reliable evidence found.

instead of:

›Inventing an answer.

Architecture:

Architecture & Data Flow
Query
 |
 v
Retrieval
 |
 +--> Strong evidence --> Generate
 |
 +--> Weak/no evidence --> Abstain / clarify

62. Data Freshness

Reliability also means using current information.

Monitor:

Architecture & Data Flow
Source update
 |
 v
Ingestion delay
 |
 v
Index update
 |
 v
Available to AI

Define freshness SLOs for time-sensitive systems.

63. Agent Reliability

Agents introduce additional failure modes:

  • infinite loops
  • incorrect plans
  • wrong tools
  • invalid arguments
  • repeated actions
  • stale state
  • unexpected side effects

Use bounded execution.

64. Agent Execution Limits

Example:

🐍 Python
limits = {
 "max_steps": 10,
 "max_tool_calls": 6,
 "max_runtime_seconds": 60,
 "max_tokens": 16000,
}

The orchestrator should enforce these limits.

65. Agent State Checkpoints

Long-running agents should persist state.

Architecture & Data Flow
Step 1
 |
 v
Checkpoint
 |
 v
Step 2
 |
 v
Checkpoint
 |
 v
Step 3

If a worker crashes, resume from the latest safe checkpoint.

66. Idempotent Agent Actions

Agents may retry.

Therefore actions should be designed to avoid duplicate side effects.

Example:

›Create ticket

Use a stable operation ID:

›agent-run-123-action-004

The downstream system can reject duplicate execution.

67. Human Escalation

A reliable agent should know when to stop.

Architecture & Data Flow
Agent
 |
 +--> Confidence high --> Continue
 |
 +--> Uncertain --> Ask user
 |
 +--> High-risk --> Human approval
 |
 +--> Dependency failure --> Escalate

Stopping safely is a reliability feature.

68. Reliability State Machine

A production agent can use:

Architecture & Data Flow
START
 |
 v
PLAN
 |
 v
ACT
 |
 v
VERIFY
 |
 +--> success --> COMPLETE
 |
 +--> retryable failure --> ACT
 |
 +--> uncertain --> HUMAN
 |
 +--> budget exceeded --> STOP
 |
 +--> fatal failure --> FALLBACK

Explicit state transitions are easier to operate than unconstrained loops.

69. Dependency Failure Propagation

Example:

Architecture & Data Flow
CRM outage
 |
 v
Tool failure
 |
 v
Agent retries
 |
 v
Latency increases
 |
 v
More requests queue
 |
 v
System saturation
 |
 v
Overall outage

Reliability engineering breaks this chain using:

  • timeouts
  • circuit breakers
  • bounded retries
  • fallbacks
  • load shedding

70. Bulkheads

Bulkheads isolate resources.

Architecture & Data Flow
 AI Platform
 |
 +-----------+-----------+
 | |
 Support pool Analytics pool
 | |
 GPU/queue A GPU/queue B

A failure in one workload should not consume all capacity.

71. Tenant-Level Isolation

Multi-tenant systems should prevent one tenant from exhausting shared resources.

Controls:

  • tenant quotas
  • concurrency limits
  • separate queues
  • weighted scheduling
  • budget controls

Example:

text
Tenant A
 |
 max 100 concurrent

Tenant B
 |
 max 20 concurrent

72. Priority Scheduling

Not all workloads are equally important.

text
Priority 0
Critical production

Priority 1
Customer-facing

Priority 2
Internal interactive

Priority 3
Batch / experiments

Under pressure, scheduling should respect priority.

73. Disaster Recovery

Ask:

text
What happens if:
- a model provider fails?
- a GPU cluster fails?
- a vector DB fails?
- a region fails?
- a database is corrupted?
- identity is unavailable?

Define recovery strategies before the incident.

74. RTO and RPO

RTO#

Recovery Time Objective:

How quickly must service recover?

RPO#

Recovery Point Objective:

How much data loss is acceptable?

Example:

Mathematical Formulation
RTO = 30 minutes
RPO = 5 minutes

These requirements influence architecture and cost.

75. Disaster Recovery Architecture

Architecture & Data Flow
Primary Region
 |
 +--> AI Gateway
 +--> Model serving
 +--> RAG
 +--> Data
 |
 v
Replication
 |
 v
Secondary Region
 |
 +--> Standby capacity

Not every component needs identical active-active deployment.

76. Model Disaster Recovery

If self-hosting:

Architecture & Data Flow
Model artifact
 |
 +--> Primary registry
 |
 +--> Backup registry

Keep:

  • model weights
  • tokenizer
  • configuration
  • serving version
  • deployment manifests

recoverable.

77. RAG Disaster Recovery

Back up:

text
Source documents
Metadata
Permissions
Embedding configuration
Index configuration

The vector index should be reproducible from authoritative source data when possible.

78. Configuration Recovery

Version:

text
Prompt
Model configuration
Routing policy
Retrieval configuration
Tool schemas
Guardrails

If production fails, the system should be reproducible.

79. Incident Response

A standard process:

Architecture & Data Flow
Detect
 |
 v
Triage
 |
 v
Contain
 |
 v
Mitigate
 |
 v
Recover
 |
 v
Validate
 |
 v
Postmortem

Assign clear ownership.

80. Incident Severity

Example:

text
SEV-1
Critical customer or safety impact

SEV-2
Major degradation

SEV-3
Limited impact

SEV-4
Minor issue

Severity should reflect business impact, not technical curiosity.

81. AI-Specific Incidents

Examples:

text
Model outage
Prompt regression
RAG permission leak
Agent loop
Unexpected cost spike
Safety violation
Provider degradation
Structured output failure
Data freshness failure

Maintain runbooks for recurring incident types.

82. Runbooks

A runbook should answer:

text
How do we detect it?
What is the first action?
What should be disabled?
What logs should we inspect?
What fallback should be enabled?
How do we validate recovery?
Who owns the incident?

83. Postmortems

A useful postmortem focuses on systems, not blame.

Include:

text
Impact
Timeline
Detection
Root causes
Contributing factors
Mitigation
Recovery
What worked
What failed
Action items

84. Reliability Testing

Test failure paths intentionally.

Examples:

text
Kill model provider
Delay vector DB
Return malformed JSON
Block CRM
Inject timeout
Saturate queue
Exhaust budget
Break authentication

This is fault injection.

85. Chaos Engineering for AI

Chaos experiments can validate:

text
Provider failover
GPU failure
Database outage
Network latency
Tool failure
Queue overload
Model degradation

The goal is not to cause random damage.

The goal is:

Discover whether the system behaves safely when components fail.

86. Synthetic Failure Testing

Example:

🐍 Python
class MockModel:
 def generate(self, prompt):
 raise TimeoutError("simulated model timeout")

Then verify:

Architecture & Data Flow
Timeout
 |
 v
Retry
 |
 v
Circuit breaker
 |
 v
Fallback

87. Reliability Regression Testing

Every important change should test:

text
Availability
Latency
Quality
Safety
Tool behavior
RAG behavior
Cost

This prevents a model or prompt upgrade from silently degrading production reliability.

88. Canary Releases

Deploy to a small fraction first:

Architecture & Data Flow
100% old version
 |
 v
95% old + 5% new
 |
 v
50% old + 50% new
 |
 v
100% new

Compare:

  • errors
  • latency
  • quality
  • safety
  • cost

89. Rollbacks

A production AI system should be able to revert:

text
Model
Prompt
RAG configuration
Tool version
Gateway policy

A rollback should be faster than debugging a live production incident.

90. Feature Flags

Use flags for risky capabilities:

Mathematical Formulation
agent_write_tools = false
advanced_reasoning = true
semantic_cache = false
new_reranker = true

Feature flags allow controlled activation and rapid disablement.

91. Reliability Architecture

Architecture & Data Flow
 Users
 |
 v
 API / WAF Layer
 |
 v
 AI Gateway
 |
 +-------------------+-------------------+
 | | |
 v v v
 Auth/Policy Routing Cache
 | | |
 +-------------------+-------------------+
 |
 +------------+------------+
 | | |
 v v v
 RAG Models Tools
 | | |
 +------------+------------+
 |
 v
 Validation / Guardrails
 |
 v
 Response
 |
 v
 Observability
 |
 +------------------+------------------+
 | | |
 SRE FinOps Eval

92. Reliability Control Plane

A mature platform can centralize:

text
Health
Routing
Budgets
Policies
Feature flags
Incident controls
Model versions

Applications then inherit common reliability capabilities.

93. AI Reliability Checklist

Before production:

text
[ ] Timeouts
[ ] Retry policy
[ ] Backoff + jitter
[ ] Circuit breakers
[ ] Fallback
[ ] Rate limits
[ ] Queue limits
[ ] Load shedding
[ ] Model health checks
[ ] Tool validation
[ ] Output validation
[ ] RAG monitoring
[ ] Agent step limits
[ ] Cost limits
[ ] Observability
[ ] Incident runbook
[ ] Rollback
[ ] Disaster recovery
[ ] Security testing

94. Practical Project 1: Reliable AI Gateway

Build:

Architecture & Data Flow
Client
 |
 v
Gateway
 |
 +--> Timeout
 +--> Retry
 +--> Circuit breaker
 +--> Routing
 +--> Fallback
 +--> Rate limit
 +--> Metrics
 |
 v
Model providers

Test provider failures and verify automatic recovery.

95. Practical Project 2: AI SLO Dashboard

Create an SLO dashboard tracking:

text
Availability
P50/P95/P99 latency
Error rate
Task success
Groundedness
Tool success
Queue depth
GPU utilization
Cost

Add error-budget tracking.

96. Practical Project 3: Resilient Enterprise RAG

Build a RAG system with:

  • retrieval timeout
  • fallback search
  • no-evidence behavior
  • permission checks
  • freshness monitoring
  • citation validation

Simulate vector DB failures.

97. Practical Project 4: Reliable AI Agent

Build an agent with:

text
max_steps
max_tool_calls
max_runtime
timeouts
retry policy
idempotency
checkpointing
human escalation

Test:

  • tool failure
  • model failure
  • repeated action
  • infinite loop
  • budget exhaustion

98. Practical Project 5: AI Load Test

Create a load-testing system that varies:

text
Concurrency
Prompt length
Output length
RAG context
Tool calls
Model

Report:

text
P50
P95
P99
Error rate
Throughput
GPU utilization
Cost/request

99. Practical Project 6: AI Disaster Recovery Drill

Design and execute a recovery drill:

Architecture & Data Flow
Primary model cluster unavailable
 |
 v
Detect
 |
 v
Fail over
 |
 v
Validate
 |
 v
Restore

Measure:

  • detection time
  • recovery time
  • data consistency
  • user impact

100. Advanced Exercise 1: Reliability Budget

Design SLOs for an enterprise assistant:

Mathematical Formulation
Availability >= 99.9%
P95 <= 3 seconds
Task success >= 95%
Critical safety failures = 0

Define SLIs and error budgets for each.

101. Advanced Exercise 2: Dependency Failure Graph

Given:

Architecture & Data Flow
Application
 |
 +--> Model
 +--> Vector DB
 +--> CRM
 +--> Identity

Create a failure-propagation graph.

For each dependency define:

  • timeout
  • retry
  • fallback
  • circuit breaker
  • degradation strategy

102. Advanced Exercise 3: Multi-Region AI

Design a two-region architecture.

Requirements:

Mathematical Formulation
RTO <= 30 minutes
RPO <= 5 minutes

Explain:

  • what is replicated
  • what is active-active
  • what is standby
  • how model artifacts are recovered
  • how RAG indexes are rebuilt

103. Advanced Exercise 4: Agent Chaos Test

Inject:

text
50% tool failure
30% model timeout
10% malformed tool results

Determine whether the agent:

  • terminates safely
  • retries correctly
  • avoids duplicate actions
  • escalates appropriately
  • stays within cost budget

104. Advanced Exercise 5: Tail Latency

Suppose:

Mathematical Formulation
P50 = 1.0 sec
P95 = 4.0 sec
P99 = 12 sec

Investigate possible causes:

  • long contexts
  • queueing
  • tool calls
  • provider throttling
  • GPU saturation

Propose changes that reduce P99 without significantly harming P50.

105. Advanced Exercise 6: Reliability vs Cost

Compare:

text
Architecture A
Single provider
Low cost

Architecture B
Two providers
Higher cost
Automatic failover

Determine when the additional reliability is economically justified.

Use:

text
Expected business loss from outage
vs
Additional resilience cost

106. Common Mistakes

Mistake 1: Treating HTTP 200 as success#

AI correctness and workflow success matter.

Mistake 2: Retrying everything#

Retries can amplify outages and duplicate side effects.

Mistake 3: No timeout#

A dependency can consume resources indefinitely.

Mistake 4: No fallback#

A single model/provider failure can become a complete application outage.

Mistake 5: Unlimited agent loops#

Agents need explicit execution limits.

Mistake 6: Ignoring tail latency#

Average latency can hide serious user-facing delays.

Mistake 7: No load shedding#

Overloaded systems can collapse instead of degrading gracefully.

Mistake 8: No dependency map#

Unknown dependencies make incidents harder to diagnose.

Mistake 9: Monitoring only infrastructure#

AI quality and safety signals are also operational signals.

Mistake 10: No rollback#

Production AI changes must be reversible.

Mistake 11: No disaster recovery plan#

A model or vector database outage should not be discovered for the first time during an incident.

Mistake 12: No failure testing#

A system that has never experienced a controlled failure is not necessarily resilient.

107. Final Mental Model

AI SRE can be summarized as:

Architecture & Data Flow
 USER REQUEST
 |
 v
 AI GATEWAY
 |
 v
 CONTROL + ROUTING
 |
 +-------------+-------------+
 | | |
 v v v
 MODEL RAG TOOLS
 | | |
 +-------------+-------------+
 |
 v
 VALIDATE + VERIFY
 |
 v
 RESPONSE
 |
 v
 OBSERVABILITY
 |
 +------------------+------------------+
 | | |
 v v v
 QUALITY COST SAFETY
 | | |
 +------------------+------------------+
 |
 v
 SRE FEEDBACK
 |
 v
 IMPROVE / TEST / RELEASE

The deepest principle is:

Reliable AI is not simply AI that stays online. It is AI that continues to provide safe, useful, timely, authorized, and recoverable behavior when models, data, tools, networks, and infrastructure inevitably fail.

108. Key Takeaways

  1. AI reliability extends beyond uptime.
  2. Availability, correctness, quality, safety, and latency should be treated as distinct dimensions.
  3. SLOs should reflect user and business expectations.
  4. Error budgets provide a framework for balancing reliability and development velocity.
  5. AI systems have many failure domains.
  6. Every dependency should have appropriate timeouts.
  7. Retries require backoff, jitter, and clear retryable-error rules.
  8. Idempotency is critical for safe retries of write operations.
  9. Circuit breakers prevent cascading failures.
  10. Fallbacks and graceful degradation preserve useful service during outages.
  11. Queueing and backpressure protect systems under load.
  12. Load shedding can prevent total system collapse.
  13. AI latency should be decomposed into its major components.
  14. Parallel execution can reduce critical-path latency.
  15. AI observability should include logs, metrics, traces, quality, cost, and safety.
  16. RAG reliability depends on ingestion, retrieval, permissions, freshness, and generation.
  17. Agents require bounded execution, checkpoints, idempotency, and escalation.
  18. Multi-tenant systems need resource isolation as well as data isolation.
  19. Disaster recovery requires explicit RTO and RPO objectives.
  20. Chaos and fault-injection testing reveal whether resilience mechanisms actually work.
  21. Canary releases, feature flags, and rollback mechanisms make AI changes safer.
  22. A reliable AI platform assumes components will fail and designs safe behavior for those failures.

109. Knowledge Check

Question 1#

Is an AI service reliable if it returns HTTP 200 for every request?

Answer: No. Availability is only one dimension. The system must also provide sufficiently correct, useful, safe, and timely outcomes.

Question 2#

What is an SLO?

Answer: A measurable reliability or quality target for a service.

Question 3#

Why are retries dangerous?

Answer: They can amplify an outage, increase load, increase cost, and duplicate side effects if operations are not idempotent.

Question 4#

What does a circuit breaker do?

Answer: It stops repeated calls to a failing dependency and allows recovery testing before normal traffic resumes.

Question 5#

What is graceful degradation?

Answer: Intentionally reducing functionality while preserving a safe and useful experience when dependencies fail.

Question 6#

Why are P95 and P99 latency useful?

Answer: They expose tail latency that averages can hide.

Question 7#

What is backpressure?

Answer: A mechanism that prevents upstream producers from overwhelming downstream systems that cannot process work at the incoming rate.

Question 8#

Why do agents need execution limits?

Answer: Agents can loop, retry, or perform unnecessary actions, causing reliability, safety, and cost problems.

Question 9#

What should a reliable RAG system do when it finds weak evidence?

Answer: It should abstain, clarify, or escalate rather than confidently inventing an answer.

Question 10#

What is the core AI SRE principle?

Answer: Design the system around expected failures so that it continues to behave safely, usefully, and predictably under degraded conditions.

110. Course Progression

The course has now progressed from enterprise economics into reliability engineering.

Architecture & Data Flow
Enterprise Generative AI
 |
 v
AI FinOps & Cost Engineering
 |
 v
AI Reliability & SRE
 |
 v
AI Red Teaming & Security Testing
 |
 v
Future / Research AI Architectures
 |
 v
Full Generative AI Capstone

The next notebook moves into AI Red Teaming & Security Testing, covering AI threat modeling, adversarial testing, prompt injection, indirect injection, jailbreaks, data exfiltration, tool abuse, agent attacks, RAG attacks, multimodal attacks, model extraction concepts, abuse cases, automated red-team pipelines, security regression testing, attack taxonomies, risk scoring, defenses, incident response, and enterprise AI security validation.

Knowledge Checkpoint

AI Reliability & SRE Checkpoint

Q1.What is the role of a Semantic Cache (e.g. GPTCache / Redis) in high-volume production LLM gateways?
ATo return stored responses for semantically equivalent past queries (based on vector similarity above a threshold), reducing cost and delivering sub-10ms response times.
BTo cache static HTML pages.
CTo store user passwords.
DTo compress images.
Q2.How does a Circuit Breaker pattern protect AI microservice architectures during downstream LLM provider outages?
AIt detects downstream error thresholds, automatically trips 'open' to stop sending failing requests, and routes traffic to fallback fallback models or static responses.
BIt physically cuts power to the server rack.
CIt disables user logins.
DIt increases API timeouts to 10 minutes.
Q3.What is Fallback Model Tiering in production LLM gateways (e.g. LiteLLM / Portkey)?
AAttempting primary frontier models (e.g. Claude 3.5 Sonnet), and automatically degrading to faster/cheaper backup models (e.g. GPT-4o-mini / LLaMA 3.1) upon timeouts or rate limits.
BRunning 3 models and picking the longest response.
CConverting text into audio when APIs fail.
DRestarting the Linux server.
Track Your Learning

Finished studying this notebook?

Mark this guide as completed to update your course progress roadmap.

Previous Lesson
Distributed LLM Inference & GPU Engineering
Next Lesson
AI FinOps & Cost Engineering