Payment systems · Distributed backend
Maat
Kill the ledger. Lose nothing. A full payment platform on Kafka, TiDB, Redis, DynamoDB, Envoy and Kubernetes. Thirteen verified runs, from retry storms to six faults in 90 seconds. Not one yen lost or duplicated.
- ¥0
- lost or double-charged across 13 verified load and failure runs
- 150/s
- sustained with no failed requests, on one machine
- 63 ms
- payment p99 at steady load (p50 44 ms)
- 300
- of 1,000 concurrent debits succeed on a ¥300 wallet, exactly
The problem
Payment systems fail in the gaps: a lost response, a retried request, a crash between two writes. The cost of being wrong is someone’s money.
Maat treats these failures as the main design problem. Every money movement is a balanced double-entry posting, every guarantee has a test that fails when the rule is removed, and the system is load-tested and broken on purpose.
After each run, a verifier replays every intent under its original key. Every wallet must add up, the ledger must pass its invariant check, and the ledger, settlement and bank must all agree.
The path of a payment
- 01
Edge
Envoy is the only way in. It verifies RS256 tokens and HMAC-signed merchant requests, and strips client identity headers.
- 02
Payment service
Client-chosen idempotency keys are leases with fencing tokens. A write-ahead state survives a lost ledger response.
- 03
Ledger
Append-only double-entry postings over mutual TLS, with reserve-then-capture holds and one conditional balance update.
- 04
Outbox
Events are written in the same transaction as the state change, then relayed to Kafka with their trace restored.
- 05
Settlement
Daily payouts, then three-way reconciliation across the ledger, the settlement worker and the bank.
Results
- Steady0%Pass
- Peak92.7%Pass
- Hot merchant0%Pass
- Retry storm36.7%Pass
- Ledger killed38.4%Pass
- Payment service killed49.5%Pass
- Merchant service killed0%Pass
- Kafka frozen0%Pass
- DynamoDB frozen60.9%Pass
- Redis frozen0%Pass
- Bank frozen0%Pass
- Database frozen7.0%Pass
- Everything40.6%Pass
Requests can fail during a fault. A run passes only if no money was lost or duplicated.
All 13 runs
| Run | Rate | Failed | p50 / p99 ms | Verified |
|---|---|---|---|---|
| Steady | 149.9/s | 0% | 44 / 63 | Pass |
| Peak | 178.7/s | 92.7% | 355 / 2,923 | Pass |
| Hot merchant | 149.9/s | 0% | 56 / 173 | Pass |
| Retry storm | 99.9/s | 36.7% | 4 / 39 | Pass |
| Ledger killed | 99.9/s | 38.4% | 252 / 1,727 | Pass |
| Payment service killed | 82.2/s | 49.5% | 54 / 2,816 | Pass |
| Merchant service killed | 99.9/s | 0% | 47 / 366 | Pass |
| Kafka frozen | 99.8/s | 0% | 48 / 80 | Pass |
| DynamoDB frozen | 100.0/s | 60.9% | 48 / 2,378 | Pass |
| Redis frozen | 92.3/s | 0% | 44 / 1,411 | Pass |
| Bank frozen | 100.0/s | 0% | 50 / 191 | Pass |
| Database frozen | 99.9/s | 7.0% | 46 / 2,327 | Pass |
| Everything | 92.4/s | 40.6% | 248 / 2,634 | Pass |
One i5 laptop in Docker Compose, one instance of each service. PASS means every intent has one outcome, every wallet adds up, the ledger balances and settlement agrees. Failed requests measure availability, which these runs don’t optimise.
Guarantees, and the test behind each
| Guarantee | Test |
|---|---|
| A wallet is never overdrawn | 1,000 concurrent debits on ¥300: exactly 300 succeed |
| A retried call applies once | 64 concurrent posts with one key produce one entry |
| A hold has one winner | Capture, void, cancel and expiry race for 40 rounds |
| Opposing transfers don’t deadlock | 400 transfers in both directions between two wallets |
| A hot merchant doesn’t serialize writers | 8 roll-ups fold every delta exactly once |
| Journal history is append-only | The service’s own database connection is refused |
| A request can’t be altered or replayed | Changed body, path, timestamp and nonce are each refused |
Key decisions
Fail closed or fail open, by risk
Without the idempotency store, payments stop, because the risk is a double charge. Without Kafka they continue, since events can be published later. The fraud check gets 80 ms, then fails open, because fraud can be reversed.
Rules live in the database
One conditional UPDATE carries the overdraft rule and the row count is the verdict. A unique key gives QR codes single use. The service’s own user has no UPDATE or DELETE on the journal.
A rate can’t create money
Each account is in one currency, and a conversion is one entry that balances in both through a platform position. An entry whose totals agree while its currencies don’t is refused.
Keep the trace across the outbox
Events are published later, on another thread, where the request’s trace no longer exists. The outbox row stores the trace, so one payment is one trace from the edge to settlement.
Observability starts with the money
The dashboard asks in order: is the money correct, are payments fast, where is the backlog. A Watchdog alert always fires, so that silence can be told apart from a broken pager.
What broke
A fractional yen
{"amount": 1.5} created a QR code for 1 yen, because the JSON library truncated decimals by default. A malformed-request test caught it.
A request refused as a duplicate of itself
On a fresh cluster, the DynamoDB client’s own retry collided with the original claim and returned 409 REQUEST_IN_PROGRESS.
A rounded ratio
Multiplying by a rounded ratio turned ¥9,900 at 150 to the dollar into $65.99. Amounts are now computed exactly from both rates and rounded once.
What these numbers don’t prove
- All numbers come from one machine with single service instances, so they show correctness under failure and not availability. A killed service is down until it restarts.
- Past roughly 150 to 300 requests a second, most requests time out.
- Balances are play money and the bank is simulated.
- The AWS Terraform is validated but has never been applied. The fraud model is trained on simulated payments, so its accuracy says nothing about real fraud.