OrgMoverReplicationSlotWALHigh
Overview
Section titled “Overview”This alert fires when a logical replication slot held by an Org Mover move on a GitLab.com Patroni primary has kept
more than 100 GiB of WAL, or more than 25% of the cluster’s max_slot_wal_keep_size, for 10 minutes.
Org Mover moves an organization from one cell to another. The pilot is the service that coordinates a move. For each source database, a producer runs on the source cell, reads the organization’s changes from the database through siphon, a change-capture tool, and ships them to the target cell. To read those changes the move holds a replication slot on each source database, and the slot keeps at least the WAL the move has not read yet, for the whole database and not only for the organization being moved. The move’s teardown drops the slot when the move finishes or is aborted.
Contributing factors include:
- A long-running write transaction on the primary. A logical slot keeps the WAL from the start of the oldest write transaction still open, so a slot the move has fully read can still keep a lot of WAL. The move is healthy, and aborting it does not help.
- The move’s producer or siphon is down or stuck, so nothing reads the slot.
- An operator paused the move’s capture and did not resume it.
- At the end of a move, the step that shuts down its producers keeps failing. The producers stay paused until it succeeds, and abort is the only way out.
- A move was aborted or finished, but its teardown has not dropped the slot yet. The teardown needs the Org Mover pilot, so a pilot that is down holds it up. The producer and siphon keep reading the slot without the pilot.
Parts of the service affected:
- Directly: the Patroni primary in the
fqdnlabel. WAL accumulates on its data volume. - Indirectly: GitLab.com. Without
max_slot_wal_keep_sizethe WAL fills the disk and the database stops accepting writes. With it, PostgreSQL invalidates the slot at the limit and the move fails.
OrgMoverReplicationSlotInactive notifies without paging when a slot has not been read for 15 minutes, so it usually comes first when nothing reads the slot. It does not come first when the slot is still read but falls behind, or when the WAL grows past the threshold within those 15 minutes: then this alert is the first warning.
Services
Section titled “Services”- Org Mover Service
- Team that owns the service: Transport (
#g_transport) - Downstream of the Patroni Service, whose disk is at risk when this alert fires.
Metrics
Section titled “Metrics”- Metric:
pg_replication_slots_restart_lsn_bytes, forslot_type="logical"andslot_name=~"orgmover_slot_.*", on the primary only (pg_replication_is_replica == 0). - Unit: bytes. It is the distance from the current WAL position to the slot’s
restart_lsn, which is the WAL PostgreSQL keeps for the slot. - Source: postgres_exporter on the Patroni primaries, in the
gitlab-gprdandgitlab-gstgMimir tenants. - Thresholds: 100 GiB, as for
SiphonLogicalReplicationSlotLagHigh, or 25% of
max_slot_wal_keep_size, as for WalReceiverSlotRetainingTooMuchWal, whichever is lower. Sustained for 10 minutes. - Normal behaviour: not measured on GitLab.com yet. On the local test stack a healthy move’s slot stays within a few hundred MiB behind the source. Staging numbers are not recorded yet.
Alert Behavior
Section titled “Alert Behavior”- Clears once the slot is back under both thresholds, or once it is dropped or lost. A lost slot (
wal_statusreadslost) means the move cannot finish: abort it. OrgMoverReplicationSlotInactive keeps firing for a lost slot. - Silence it only for a deliberate pause of a move’s capture that has a change request or an incident, for the planned window, naming it in the silence. A deliberate pause that grows past these thresholds should normally be resumed or aborted rather than silenced.
- This should be a rare alert. A healthy move keeps its slot well under the threshold.
Severities
Section titled “Severities”s2,pager: pagerduty. It pages because the slot keeps WAL for the whole database, so a stalled move can exhaust the disk of a GitLab.com Patroni primary.- Escalate to
s1if the primary’s data volume is above 70% and still rising.
Verification
Section titled “Verification”On the primary, list Org Mover’s slots, the WAL each keeps (retained) and the WAL its move has not read yet
(unread):
SELECT slot_name, active, wal_status, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS unread FROM pg_replication_slots WHERE slot_name LIKE 'orgmover_slot_%';Recent changes
Section titled “Recent changes”Troubleshooting
Section titled “Troubleshooting”-
Rule out a long-running transaction. Run the query under Verification. If
unreadis small butretainedis large, the move reads its slot, but a long-running transaction on the primary holds the WAL. Handle it with postgres-long-running-transaction. Do not abort the move for it. -
Find the move. The slot name is
orgmover_slot_plus a hash of the move, its service and its database, so it does not say which organization is moving. A move holds one slot for each source database, so you can see more than one slot for the same move. Today, search the source cell’s producer logs for the slot name: when a move’s capture starts, the producer logssiphon producer createdwith the slot and the move’s id intransfer_id. In gstg, this query searches thegl-org-mover-gstglogging project for that line: add the slot name to it. If you cannot find it, ask in the#g_transportSlack channel. The capture row in the Org Mover GUI and the log line for a dropped slot will name each slot and its publication once https://gitlab.com/gitlab-org/cells/org-mover/org-mover/-/merge_requests/374 is merged. -
Check why it is not read. On the capture row, see whether the move’s capture is paused or its producer is failing or halted, and check that the source cell is running. If the move is already aborted or finished and the slot is still there, check that the Org Mover pilot is running: the teardown that drops the slot needs it.
-
Resume, retry or abort.
- The move should go on: resume its capture, or fix its producer. The slot then drains.
- The producer halted because the data broker refused a captured change for storage: a stream is at its cap, or the broker’s store is full. Raise the full one, then take Retry on the capture row. See Broker storage. If the slot is already lost, Retry is refused and the move has to be aborted.
- The move should stop: abort it. The abort’s teardown drops the slot and its publication.
- The producer service on the source cell is down: start it first. An abort drops the slot through the producer, and waits until the producer answers.
- The producer service is up: do not restart it while any move runs on it. A restarted producer forgets every move it holds, and a move that has not cut over yet can then only be aborted.
-
Do not drop the slot of a live move by hand. If the move is still running, it does not notice the drop: the changes written on the source while the slot was gone never reach the target, and nothing reports it. Drop a slot by hand only when no move holds it any more (its move is aborted or finished and the teardown did not drop it), and only as a last resort before the disk fills:
SELECT pg_drop_replication_slot('<slot_name>');
Possible Resolutions
Section titled “Possible Resolutions”- No prior incidents recorded yet; this alert is new.
Dependencies
Section titled “Dependencies”- The Org Mover pilot and the source cell’s producer for the move.
- The Patroni primary of the affected cluster, which owns the slot.
Escalation
Section titled “Escalation”- Primary contact:
#g_transportSlack channel. - If the disk is at risk and there is no response, escalate to
#g_database_operations.
Definitions
Section titled “Definitions”- Alert definition (Jsonnet source)
- Generator wiring
- Org Mover deployment guide, known limits
- Edit this playbook