Checkout was unavailable for twenty-six minutes on Friday evening. The outage was caused by a config change that had been deployed at 14:52 without a staging check. The problem was noticed ten minutes later, when an error-rate alert was triggered. The gateway was ruled out immediately, and the cause was narrowed down to a single environment variable. The release was rolled back at 15:20, and the service was fully restored by 15:24. Roughly four per cent of payments were affected; all of them have since been retried. Two actions have been agreed: the staging check has been made mandatory, and alerting will be broken down by service before 30 June.

Olha, 15:04 — Heads-up: checkout went down about ten minutes ago. I’m looking into it.
Taras, 15:06 — Is that the config we deployed at three? It never went through staging.
Olha, 15:09 — Looks like it. I’ve ruled out the gateway — it’s a single variable.
Marta, 15:11 — Support is getting asked about it already. I’ll tell them we’re on it.
Olha, 15:20 — Rolled it back. Checkout’s up.
Taras, 15:21 — Thank goodness. I’ll make staging mandatory so we don’t get burned by this again.