OrgMoverReplicationSlotInactive
Overview
Section titled “Overview”This alert fires when a logical replication slot held by an Org Mover move on a GitLab.com Patroni primary has not been read for 15 minutes.
Org Mover moves an organization from one cell to another. The pilot is the service that coordinates a move. For each source database, a producer runs on the source cell, reads the organization’s changes from the database through siphon, a change-capture tool, and ships them to the target cell. The producer reads those changes through a replication slot on each source database, and a slot nobody reads keeps WAL for the whole database until it is read again, dropped or lost.
Contributing factors include:
- An operator paused the move’s capture. This is expected, and the reason this alert does not page.
- The move’s producer or siphon is down or keeps failing, or the producer halted on a data broker refusal.
- PostgreSQL invalidated the slot at
max_slot_wal_keep_size(wal_statusreadslost). Nothing can read it any more, so the move cannot finish: abort it. - A move was aborted or finished, but its teardown has not dropped the slot yet. The teardown needs the Org Mover pilot, so a pilot that is down holds it up. The producer and siphon keep reading the slot without the pilot.
A slot also reads inactive for a few seconds while a siphon restarts, and while a move pauses its producers before tearing them down at its end. That pause is usually brief and the 15-minute wait lets it pass. But when the step that shuts down the producers at the end of a move keeps failing, they stay paused for as long as it fails, and abort is the only way out. So this alert can be a stuck move, not only a short pause.
Parts of the service affected:
- Directly: the Patroni primary in the
fqdnlabel, which keeps WAL for the slot. - If the WAL keeps growing, OrgMoverReplicationSlotWALHigh pages.
Services
Section titled “Services”- Org Mover Service
- Team that owns the service: Transport (
#g_transport) - Downstream of the Patroni Service.
Metrics
Section titled “Metrics”pg_replication_slots_active == 0, forslot_type="logical"andslot_name=~"orgmover_slot_.*", on the primary only (pg_replication_is_replica == 0).- Source: postgres_exporter on the Patroni primaries, in the
gitlab-gprdandgitlab-gstgMimir tenants. - It does not check the WAL the slot keeps, so a lost slot, whose
restart_lsnis empty, keeps firing.
Alert Behavior
Section titled “Alert Behavior”- Fires after 15 minutes, the same wait as PatroniClusterPaused, so a short pause or a producer restart can finish first. Clears as soon as the slot is read again or dropped. A lost slot keeps it firing until the move is aborted and its teardown drops the slot.
- Silencing is right for a deliberate pause of a move’s capture: silence it for the planned window, matching the
slot_name, and name the change request or the move in the silence. While it is silenced, OrgMoverReplicationSlotWALHigh still pages if the slot keeps too much WAL.
Severities
Section titled “Severities”s3, no page. A paused or stalled move costs only WAL disk at first, and the WAL-size alert pages before that becomes a risk.
Verification
Section titled “Verification”On the primary, list Org Mover’s slots, the WAL each keeps (retained) and the WAL its move has not read yet
(unread):
SELECT slot_name, active, active_pid, wal_status, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS unread FROM pg_replication_slots WHERE slot_name LIKE 'orgmover_slot_%';A slot keeps the WAL from the start of the oldest write transaction still open on the primary. When unread is
small but retained is large, a long-running transaction holds the WAL, not the move: see
postgres-long-running-transaction.
Recent changes
Section titled “Recent changes”Troubleshooting
Section titled “Troubleshooting”- Find the move. The slot name is
orgmover_slot_plus a hash of the move, its service and its database, so it does not say which organization is moving. A move holds one slot for each source database, so you can see more than one slot for the same move. Today, search the source cell’s producer logs for the slot name: when a move’s capture starts, the producer logssiphon producer createdwith the slot and the move’s id intransfer_id. In gstg, this query searches thegl-org-mover-gstglogging project for that line: add the slot name to it. If you cannot find it, ask in the#g_transportSlack channel. The capture row in the Org Mover GUI and the log line for a dropped slot will name each slot and its publication once https://gitlab.com/gitlab-org/cells/org-mover/org-mover/-/merge_requests/374 is merged. - Paused on purpose: acknowledge, silence for the planned window as above, and check with whoever paused it that they plan to resume or abort.
- Not paused: resume, retry or abort. Check that the source cell is running, and whether the capture row shows
the producer failing or halted. If the move is already aborted or finished and the slot is still there, check
that the Org Mover pilot is running: the teardown that drops the slot needs it. Then:
- The move should go on: resume its capture, or fix its producer. The slot then drains.
- The producer halted because the data broker refused a captured change for storage: a stream is at its cap, or the broker’s store is full. Raise the full one, then take Retry on the capture row. See Broker storage. If the slot is already lost, Retry is refused and the move has to be aborted.
- The move should stop: abort it. The abort’s teardown drops the slot and its publication.
- The producer service on the source cell is down: start it first. An abort drops the slot through the producer, and waits until the producer answers.
- The producer service is up: do not restart it while any move runs on it. A restarted producer forgets every move it holds, and a move that has not cut over yet can then only be aborted.
- Do not drop the slot of a live move by hand. If the move is still running, it does not notice the drop: the changes written on the source while the slot was gone never reach the target, and nothing reports it. Drop a slot by hand only when no move holds it any more.
Possible Resolutions
Section titled “Possible Resolutions”- No prior incidents recorded yet; this alert is new.
Dependencies
Section titled “Dependencies”- The Org Mover pilot and the source cell’s producer for the move.
- The Patroni primary of the affected cluster, which owns the slot.
Escalation
Section titled “Escalation”- Primary contact:
#g_transportSlack channel.
Definitions
Section titled “Definitions”- Alert definition (Jsonnet source)
- Generator wiring
- Org Mover deployment guide, known limits
- Edit this playbook