Forward-deployed / Learning zone
System Designa standalone module
Lesson 08

Transactional & financial systems

TL;DR

Financial and transactional systems are where distributed systems meet zero tolerance for error. A hotel reservation system prevents double-booking through database constraints and idempotency keys on a modest 3 TPS write load. A gaming leaderboard exploits Redis sorted sets for O(log N) rank operations across 5M daily players. A payment system orchestrates PSP integration, double-entry ledgers, and nightly reconciliation to ensure every cent is accounted for. A digital wallet evolves from naive Redis to event-sourced Raft replication as TPS demands grow from thousands to millions. A stock exchange pushes latency to microseconds with single-server mmap'd memory, custom sequencers, and reliable UDP multicast. The common theme: correctness is more important than availability. The cost of a bug is measured in dollars, not just user frustration.

🎯 For the technical PM

Why it matters — These systems handle money, reservations, and competitive rankings — domains where a bug is a financial loss, a legal liability, or a PR disaster. The engineering tradeoffs are dominated by correctness constraints that don't exist in social media or content systems.

What it changes in your decisions — You accept higher latency for stronger consistency. You insist on idempotency for every write operation. You budget for reconciliation infrastructure and audit trails. You question every "eventually consistent" proposal when money is involved.

Ask your eng team — "What happens if this operation is executed twice — do we double-charge, double-book, or is it safely idempotent?"

Risk if ignored — You double-book hotel rooms during peak season, double-charge customers during payment retries, or discover that your ledger is off by $2M during the quarterly audit.

Hotel reservation system

Scale and requirements

Data model: reserve roomTypeID, not roomID

A critical design insight: guests reserve a room type, not a specific room. Room assignment happens at check-in, not at booking time.

Transactional and financial systems
Five systems, four shared patterns that make money safe
ACID Transactionscorrectness Idempotencysafe retries Event Sourcingaudit trail Reconciliationconsistency check
1
Payment System
PSP integration, ledger
Key mechanism
Double-entry ledger: every payment creates a debit + credit row that must sum to zero. Idempotency key prevents duplicate charges on retry.
ACID Idempotency Events Reconcile
2
Digital Wallet
Balance management
Key mechanism
Event-sourced balance: current balance is SUM(events), never stored directly. Transfer = atomic debit from wallet A + credit to wallet B in one transaction.
ACID Events Reconcile
3
Stock Exchange
Matching engine
Key mechanism
Order book with price-time priority. Matching engine processes orders sequentially on a single thread for determinism. Sequencer assigns global order IDs.
ACID Events
4
Auction System
Real-time bidding
Key mechanism
Optimistic locking on current_price: bid accepted only if bid > version_price. WebSocket push for live price updates. Escrow holds funds until auction closes.
ACID Idempotency
5
Ticket Booking
Inventory + reservations
Key mechanism
Two-phase reservation: HOLD (TTL) then CONFIRM or expire. SELECT ... FOR UPDATE prevents double-booking the same seat.
ACID Idempotency Reconcile
The universal safety net
Every request carries an idempotency_key → checked before processing → if seen, return cached result
Every state change emits an event → append to immutable log → nightly reconciliation job diffs events vs. balances → alerts on mismatch

This simplification means the availability check is a count query (SELECT available_count FROM inventory WHERE hotel_id=X AND room_type='KNG' AND date BETWEEN ...), not a scan of individual room status.

Double-booking prevention

Three approaches, with increasing sophistication:

Pessimistic locking:

SELECT ... FOR UPDATE WHERE hotel_id=X AND room_type='KNG' AND date='2024-10-15'
-- Check availability
-- If available: UPDATE SET available_count = available_count - 1
COMMIT

The FOR UPDATE lock blocks all other transactions trying to book the same room type on the same date. Safe but serializes all bookings — a bottleneck at high concurrency.

Optimistic locking (version column):

SELECT available_count, version FROM inventory WHERE ...
-- Check availability (application side)
UPDATE inventory SET available_count = available_count - 1, version = version + 1
  WHERE ... AND version = {read_version}
-- If 0 rows updated: conflict, retry

No locks held during the check. If two transactions read the same version, only one succeeds. The other retries. Better concurrency, but retries add latency under contention.

Database constraints (preferred):

ALTER TABLE inventory ADD CONSTRAINT chk_availability
  CHECK (available_count >= 0);

UPDATE inventory SET available_count = available_count - 1
  WHERE hotel_id=X AND room_type='KNG' AND date='2024-10-15';
-- DB rejects if available_count would go negative

The database enforces the invariant. No application-level locking logic. The simplest correct solution for this scale.

Idempotency via reservationID

Network failures cause retries. Without idempotency, a retry creates a duplicate reservation:

  1. Client generates a unique reservation_id (UUID) before sending the request.
  2. Server uses reservation_id as the primary key (or a unique constraint).
  3. If the same reservation_id arrives twice, the second INSERT is rejected (duplicate key) and the server returns the existing reservation.

This pattern is universal in financial systems — it appears in every design in this lesson.

Why MySQL (ACID) at this scale

At ~3 TPS, there is no need for distributed databases. A single MySQL instance with replication handles this comfortably:

The lesson: don't reach for distributed databases when a well-tuned RDBMS solves the problem.

Gaming leaderboard

Scale and requirements

Redis sorted sets

Redis sorted sets are purpose-built for this:

Redis sorted sets

Increment, rank, and top-N — all O(log N)

At 25M members, that's ~25 comparisons — sub-millisecond on Redis.

Game server
→
Redis sorted set — leaderboard:2024-10
←
Client
ZINCRBY +50 points
ZREVRANK — my rank?
ZREVRANGE 0 9 — top 10?

All three core operations — increment, rank lookup, and top-N — are O(log N). At 25M members, that means about 25 comparisons — sub-millisecond on Redis.

Monthly keys and TTL

Each time period gets its own sorted set key:

Monthly keys get a TTL (e.g., 90 days after the month ends). This avoids unbounded memory growth.

Scaling beyond a single Redis instance

A single Redis instance holds the sorted set in memory. For 25M members with 8-byte scores and ~20-byte member names, that's ~700 MB — well within a single instance's capacity.

But if the player base grows to hundreds of millions, or if there are thousands of concurrent leaderboards:

Range partitioning — split the score range across Redis instances:

Problem: score distributions are skewed (most players cluster in the middle), so partitions are uneven.

Hash partitioning — hash player_id to a shard. Each shard holds a partial leaderboard.

For most games (under 100M players), a single beefy Redis instance is sufficient and simpler.

Payment system

Scale and requirements

PSP integration and hosted payment page

The payment system does not handle raw credit card numbers. Instead:

  1. The client loads a hosted payment page from the PSP (Stripe Checkout, Braintree Drop-in).
  2. The user enters card details directly into the PSP's iframe.
  3. The PSP returns a payment token to the client.
  4. The client sends the token to the merchant's backend.
  5. The backend calls the PSP API with the token to execute the charge.
PSP integration

The merchant never sees a card number

The hosted payment page keeps you out of PCI scope entirely.

User
Client app
PSP — Stripe
Merchant backend
1Client → PSP: load hosted payment page
2User → PSP: enter card details
3PSP → Client: payment token (tok_xxx)
4Client → Merchant: submit order + token
5Merchant → PSP: charge token (amount, idempotency_key)
6PSP → Merchant → Client: payment result, order confirmation
This keeps the merchant out of PCI scope — they never see, store, or transmit card numbers.

This keeps the merchant out of PCI scope — they never see, store, or transmit card numbers.

Double-entry ledger

Every money movement is recorded as two entries that sum to zero:

Transaction Debit (from) Credit (to) Amount
Customer pays Customer wallet Merchant revenue $100.00
PSP fee Merchant revenue PSP fees payable $2.90
Refund Merchant revenue Customer wallet $100.00

The invariant: sum of all debits = sum of all credits, always. If they diverge, something is wrong and the discrepancy must be investigated before any more transactions process.

Double-entry bookkeeping is 700 years old and remains the gold standard because:

Idempotency via payment_order_id

The most dangerous failure in payments: a network timeout after the PSP charges the card but before the merchant records the success. The merchant retries, and the customer is charged twice.

Prevention:

  1. The merchant generates a unique payment_order_id before calling the PSP.
  2. This ID is sent as the PSP's idempotency_key.
  3. If the PSP receives the same idempotency_key twice, it returns the original result without re-executing.
  4. The merchant's own database uses payment_order_id as a unique key.

Retry queue and dead-letter queue

Failed payment attempts follow a structured retry path:

Retry queue & dead-letter queue

Transient vs. permanent failures get different treatment

Card declined never retries. A timeout retries with exponential backoff.

Payment request → Execute via PSP
↓
Success → Record in ledger
Transient failure (timeout, 500) → Retry queue, backoff
Permanent failure (declined) → Mark failed
Retry queue → max retries exceeded → Dead-letter queue (manual investigation)

Nightly reconciliation

The payment system, PSP, and bank each have their own record of every transaction. Reconciliation verifies that all three agree:

  1. Download the PSP's settlement report (all charges, refunds, fees for the day).
  2. Download the bank's statement (deposits received).
  3. Compare against the internal ledger, line by line.
  4. Flag discrepancies for investigation.

Discrepancies happen more often than you'd expect: timezone differences, currency rounding, PSP processing delays, partial refunds. A robust reconciliation pipeline is not optional — it's how you catch errors before the quarterly audit.

Digital wallet

Scale and requirements

Evolution of the architecture

The architecture evolves through four stages as TPS requirements grow:

Stage 1: Redis (simple but fragile)

Store balances in Redis as key-value pairs. INCR/DECR for transfers. Fast but no durability guarantee — a Redis crash loses uncommitted transactions.

Stage 2: Sharded RDBMS with 2PC

Move balances to MySQL, sharded by user_id. A transfer between two users on different shards requires two-phase commit (2PC):

  1. Prepare: both shards lock the rows and confirm they can execute.
  2. Commit: the coordinator tells both shards to commit.

If either shard fails during prepare, both abort. If the coordinator crashes after prepare but before commit, the shards remain locked until the coordinator recovers. This is the blocking problem of 2PC.

Try-Confirm/Cancel (TC/C) is a business-level alternative:

  1. Try: tentatively debit the sender (hold the amount).
  2. Confirm: credit the receiver and finalize the debit.
  3. Cancel: if confirm fails, reverse the hold.

TC/C avoids distributed locks but requires compensating transactions (reversal logic).

Stage 3: Event sourcing + CQRS

Event sourcing + CQRS

The balance is a projection, not a column

To fix a wrong transaction, append a reversal — never modify history.

Transfer command
→
Event store — append-only log
→
Projection — materialize balances
→
Read model — current balances
Event store → Audit log — complete history
Writes: event store · Reads: read model — CQRS

Instead of updating a balance column, append an event: {type: "transfer", from: A, to: B, amount: 50, timestamp: ...}.

The current balance is a projection — computed by replaying all events for that user. This is cached in a read model (Redis or a materialized view) for fast lookups.

CQRS (Command Query Responsibility Segregation): writes go to the event store. Reads come from the projected read model. They can scale independently.

Stage 4: Raft replication

At 1M TPS, even sharded databases struggle. The final architecture uses a custom storage engine with Raft consensus for replication:

Why event sourcing for wallets

Event sourcing is not just an implementation choice — it's a regulatory requirement:

Stock exchange

Scale and requirements

Single-server architecture with mmap

The most surprising design choice: the matching engine runs on a single server, not a distributed cluster.

Why single-server:

mmap'd shared memory connects components within the server:

Single-server mmap architecture

In-process nanoseconds beat network microseconds

Shared memory means zero-copy handoffs between components.

Gateway — network I/O
↓
Single server
Sequencer — assign order
mmap →
Order book — matching engine
mmap →
Trade reporter
↓
Market data publisher

mmap allows multiple processes to share a memory region without serialization/deserialization overhead. The sequencer writes an order to shared memory. The matching engine reads it directly — zero copy.

Custom sequencer (not Kafka)

A stock exchange cannot use Kafka as a sequencer because:

The custom sequencer:

  1. Receives orders from the gateway.
  2. Assigns a globally monotonic sequence number.
  3. Writes the sequenced order to the mmap'd ring buffer.
  4. The matching engine consumes from the ring buffer in sequence order.

Order book: doubly-linked list + hash map

The order book tracks all outstanding buy and sell orders for a symbol, sorted by price-time priority:

Order book: price-time priority

Doubly-linked list of price levels + hash map by order_id

Add, cancel, and match are all O(1) — the constant factor beats tree traversal at microsecond latency.

Order book for AAPL
Asks — ascending price
$150.10 × 200 shares
$150.15 × 500 shares
$150.20 × 300 shares
Bids — descending price
$150.05 × 400 shares
$150.00 × 600 shares
$149.95 × 100 shares

Data structure: a doubly-linked list of price levels, with a hash map from order_id to its node in the list:

Why not a tree? Trees have O(log N) operations. At microsecond latency targets, the constant factor of hash map + linked list beats tree traversal. Cache locality matters — the linked list at each price level keeps related orders contiguous in memory.

Event sourcing + Raft for durability

The exchange uses event sourcing: every order, cancellation, and trade is an event in an append-only log. The order book state is a deterministic projection of this log.

For disaster recovery, the event log is replicated via Raft consensus to standby servers:

Reliable UDP multicast for market data

Market data (price updates, trade executions) must reach thousands of subscribers simultaneously. TCP unicast doesn't scale — one connection per subscriber, with TCP's congestion control adding latency.

Reliable UDP multicast:

The tradeoff: UDP multicast requires network infrastructure support (IGMP, switches with multicast routing). It works within a datacenter or across connected exchanges, not over the public internet.

Failure modes

Practitioner checklist