Skip to content
Past Localhost
M02State and side effectsLesson 4 of 4

M02 summary and postmortem: Twilio

The core ideas on state and idempotency, review questions and the incident where a read-only Redis kept recharging credit cards.

What to take from this module

Section titled “What to take from this module”
  • State is spread out: client, CDN, process memory, Redis, database, queues and external providers. For each one, ask: is it shared? does it survive a restart? who can change it?
  • Process memory is disposable. Stateless processes let you run several instances, deploy and autoscale without losing anything.
  • Every important piece of data has one source of truth. The other copies are derived and may be stale.
  • Over a network, any operation can run more than once. Make everything that’s expensive to repeat idempotent.
  • Four techniques: absolute operations, a natural key with ON CONFLICT, conditional transitions and idempotency keys.
  • Infrastructure repeats by design: retries, queues, webhooks and cron on several instances. That’s why the receiver has to tolerate duplicates.

Postmortem

Section titled “Postmortem”

Review questions

Section titled “Review questions”
Your app keeps the shopping cart in an in-memory Map. What three problems does that cause in production?

Each instance sees a different cart (it isn’t shared), the cart disappears on every deploy or crash (it doesn’t survive a restart) and the Map grows without limit (a memory leak). Move it to Redis or the database, with expiry.

What’s the difference between “idempotent” and “side-effect free”?

A side-effect-free operation changes nothing, like a read. An idempotent operation changes state on the first run and nothing more on the runs after it, like SET plan = 'pro'.

Why should the idempotency key be generated on the client and not on the server?

Because the point is to recognize retries of the same intent. If the server generates the key, each retry arrives as a new request and gets a new key. Only the client knows that the second request repeats the first.

A webhook handler writes the event ID to a table and then applies the effect, as two separate operations. What failure does that allow?

If the process dies between the two operations, the event is marked as processed without the effect having been applied. The provider’s retry gets ignored and the effect is lost. Both operations need to be in the same transaction.

With three instances, how do you make sure a scheduled job runs only once?

With a distributed lock, such as pg_try_advisory_lock in Postgres, or with a single scheduler separate from the web servers. The job should also be idempotent, in case it reruns after a failure.

Go deeper

Materials in English unless noted.

Written by About this guide