System Design Diagnostic Playbook
In Lesson 1 twe learned about the foundational Primitives (Load Balancers, Caches, Databases, Queues). In Lesson 2 introduced the 3-Layer Architectural Framework.
In Lessons 3 & 4 demonstrated these principles in action through real-world case studies: spatial ride-matching with Bolt, recommendation engines with Spotify, and RAG retrieval/streaming with ChatGPT along with the trade offs when it comes to building real systems.
Now, we bring it all together. When faced with an open-ended system design prompt—whether in an interview or on the job—you don't start by picking tech stacks. You start by asking the right questions to uncover constraints, select the right primitives, map out the architectural layers, and evaluate the necessary trade-offs.
So in this playbooks, I give the questions to be answered when building a system from scratch and you also get a prompt to give to any AI system to help you develop your system of any complexity or scale.
Step 1: Interface & Execution Constraints (Latency vs. Throughput)
Q1.1: Who or what is consuming the output of this system?
If an active human user on an interactive screen/voice interface:
Mapping: Target Sub-100ms Low Latency.
Example: An interactive AI voice agent streaming back synthesized audio character-by-character over WebRTC.
If a background process, worker queue, or analytics engine:
Mapping: Target High Throughput Batching.
Example: A nightly analytics pipeline generating sentiment labels across millions of customer feedback tickets.
Q1.2: How long can the client wait for a complete response before the operation fails or degrades user experience?
If 300ms (e.g., autocomplete, voice AI, ad bidding):
Communication Layer Mapping: Use WebSockets / Server-Sent Events (SSE) or low-latency gRPC.
Structural Layer Mapping: Use fast, localized CPU/GPU workers with pre-warmed connection pools.
Example: An IDE inline code-completion engine suggesting the next line of code as a developer types.
If seconds, minutes, or hours (e.g., bulk document indexing, dataset labeling, video rendering):
Communication Layer Mapping: Use Asynchronous Message Queues (Kafka, RabbitMQ, Celery).
Structural Layer Mapping: Use dynamic worker pools with continuous batching.
Example: An enterprise RAG ingestion engine embedding 500,000 PDF documents into a vector database.
Step 2: Data Integrity & Failure Behavior (Consistency vs. Availability)
Q2.1: If a network partition occurs between database nodes, what is the absolute worst outcome?
If financial loss, double-spending, overselling inventory, or security breaches occur:
Mapping: Choose Strong Consistency (CP System).
Primitive Mapping: Use Relational SQL Databases (PostgreSQL, CockroachDB, Spanner) with ACID transactions and synchronous commits.
Example: A banking transaction engine processing account withdrawals or a concert ticketing app locking seat reservations.
If users see slightly outdated like counts, delayed comments, or stale status indicators:
Mapping: Choose High Availability (AP System).
Primitive Mapping: Use NoSQL / Distributed Key-Value Stores (Cassandra, DynamoDB) with Eventual Consistency and asynchronous replication.
Example: A social media platform updating view counts, likes, and comments across global user feeds.
Q2.2: How fast does a state change in Step A need to be visible to Step B globally?
Instantaneous across all nodes:
Mapping: Synchronous multi-phase commits (sacrifices write latency).
Example: A high-frequency trading platform verifying account balances before executing a trade.
Eventually consistent (\(1\text{--}3\) seconds delay acceptable):
Mapping: Local-first reads with background message queue synchronization.
Example: A user updating their profile picture or active "Online" status badge.
Step 3: Infrastructure Economics & SLA Targets (Cost vs. Performance)
Q3.1: What is the financial or business cost of 10 minutes of complete system downtime?
Severe (Millions in lost revenue, legal SLA penalty fees):
Mapping: Target High Performance / \(99.999\%\) Availability ("Five Nines").
Structural Mapping: Multi-region active-active deployments, redundant load balancers, over-provisioned auto-scaling groups, and dedicated hot standby clusters.
Example: A primary payment gateway like Stripe or Visa handling core checkout flows for global enterprise clients.
Minimal (Early MVP, internal admin tool, side project):
Mapping: Target Cost-Optimized / \(99.9\%\) Availability.
Structural Mapping: Single-region deployment, single primary database with automated backups, serverless compute, and basic auto-scaling.
Example: An internal employee dashboard or pre-revenue startup MVP validating product features.
Q3.2: Does the data access pattern hit a small subset of "hot" data frequently?
Yes:
Primitive Mapping: Implement an In-Memory RAM Cache Layer (Redis / Memcached) or GPU VRAM KV Caching to bypass expensive disk reads.
Example: A streaming platform serving top trending tracks or a multi-turn chat application caching past conversation tokens.
No (Uniform random reads):
Primitive Mapping: Scale out database read-replicas and disk I/O bandwidth.
Example: An archival document platform performing cold search queries across historical logs.
Step 4: The 3-Layer Architectural Mapping
Based on your answers to Steps 1–3, complete your system mapping:
Layer 1: Structural Pattern (Compute Isolation)
Choice: Monolith vs. Microservices / Decoupled Nodes.
Selection Criteria: Separate CPU-bound business logic from GPU inference or high-memory background pipelines.
Example: Decoupling ChatGPT's FastAPI web API microservices from dedicated vLLM GPU inference clusters.
Layer 2: Communication Pattern (Data Routing)
Choice: Synchronous (HTTP/gRPC) vs. Asynchronous (Kafka/Queues) vs. Streaming (SSE/WebSockets).
Selection Criteria: Governed by Q1.1 and Q1.2 latency thresholds.
Example: Using HTTP gRPC for vector lookups while using Server-Sent Events (SSE) to stream output tokens.
Layer 3: Persistence Scaling (Storage & Caching)
Choice: Relational SQL vs. NoSQL / Vector DB + RAM Caching Strategy.
Selection Criteria: Governed by Q2.1 consistency requirements and Q3.2 data access patterns.
Example: Storing user profiles in PostgreSQL, raw text chunks in AWS S3, document vectors in Qdrant RAM, and prompt history in GPU VRAM KV cache.
System Architect Meta-Prompt
Copy and paste the text below into any advanced LLM (ChatGPT, Claude, Gemini) to transform it into an interactive System Architect.
START OF PROMPT
SYSTEM INSTRUCTION: SYSTEM ARCHITECT INTERACTIVE BOT
You are an elite Principal Systems Architect. Your objective is to help the user design a production-grade, highly scalable system architecture for any software idea or feature.
STEP 1: INITIAL DISCOVERY & DIAGNOSTIC QUESTIONNAIRE
First, ask the user to briefly describe what their system is aimed at doing (its primary purpose, business goals, and core features).
Then, present the following 5 diagnostic questions to gather the necessary constraints. Instruct the user to answer them as best as they can:
Interaction Style & Latency: Who or what is consuming the output, and what is the target latency threshold?
Option A: Active human user expecting an interactive screen/voice response (<300ms latency required).
Option B: Background worker, queue, or batch analytics engine (seconds/hours acceptable).
Data Integrity vs. Availability (CAP Theorem): If a network partition occurs between database nodes, what is the worst acceptable outcome?
Option A: Fail fast/Block writes to guarantee 100% data correctness (Financial/Inventory lock).
Option B: Stay online 100% of the time, accepting slightly stale data temporarily (Social feeds/View counts).
State Propagation: How fast must a state change in Step A be visible globally to Step B?
Option A: Instantaneous across all nodes (blocking synchronous commits).
Option B: Eventually consistent (1–3 second delay is acceptable).
Infrastructure Economics & Scale: What is the financial/business cost of 10 minutes of complete system downtime?
Option A: Severe (Millions lost / strict legal SLAs) -> Requires 99.999% uptime multi-region active-active deployment.
Option B: Minimal (MVP / Internal tool) -> Requires low-cost single-region deployment.
Data Access Patterns: Does your workload frequently access a small subset of "hot" data?
Option A: Yes (Requires aggressive in-memory RAM/VRAM caching).
Option B: No (Uniform random reads across disk storage).
STEP 2: OUTPUT GENERATION
Once the user describes their system goals and answers the questions, generate a comprehensive, publication-ready System Architecture Blueprint structured into the following sections:
SECTION 1: SYSTEM OVERVIEW & QUALIFYING CONSTRAINTS
A concise summary of the task and core business objectives.
A explicit list of identified system constraints (Latency target, CAP theorem preference, SLA targets, and Caching requirements).
SECTION 2: THE 3-LAYER ARCHITECTURAL BREAKDOWN
Layer 1: Structural Pattern (Compute Isolation)
- Detail how compute nodes are separated (e.g., decoupling lightweight CPU web services from heavy background workers, spatial engines, or GPU inference clusters).
Layer 3: Communication Pattern (Data Movement)
- Map exact protocols to use (e.g., Synchronous gRPC/HTTP, Asynchronous Message Queues like Kafka, or Real-Time Streaming via SSE/WebSockets).
Layer 3: Persistence & Scaling (Storage & Memory)
- Define storage layers (Relational SQL, NoSQL key-value, Vector Databases, Blob storage) and caching strategies (Redis, CDN, GPU VRAM KV cache).
SECTION 3: RECOMMENDED TECH STACK & EXECUTION TOOLS
Provide a tailored table listing the best-in-class production tools for executing this system:
| Layer / Component | Recommended Tool / Technology | Technical Rationale |
|---|---|---|
| Compute / Runtime | [e.g., Node.js, Go, FastAPI, Ray] | [Why it fits the latency/throughput constraint] |
| Primary Database | [e.g., PostgreSQL, CockroachDB, DynamoDB] | [Why it fits the CP/AP requirement] |
| Caching Layer | [e.g., Redis, Dragonfly, In-Memory H3] | [Why it fits the hot data pattern] |
| Messaging / Streaming | [e.g., Apache Kafka, RabbitMQ, SSE, WebSockets] | [Why it fits the communication model] |
| Infrastructure / Cloud | [e.g., AWS EKS, Cloudflare Workers, vLLM Engine] | [Why it fits the budget/SLA constraint] |
SECTION 4: SYSTEM ARCHITECTURE FLOWCHART
Provide a visual Mermaid.js diagram showing data flow from Client -> Compute Layers -> Messaging -> Persistence/Caching Layers.
SECTION 5: TRADE-OFF ANALYSIS MATRIX
Provide a summary table detailing:
Primary Metric Optimized (e.g., Sub-100ms Latency, High Batch Throughput, Strong Consistency, Low Cost).
Intentionally Sacrificed Metric.
Technical justification for why this trade-off is optimal for the user's business goals.
Begin by asking the user to describe their system goal and presenting the 5 diagnostic questions.
END OF PROMPT
