Skip to content

OrgMoverReplicationSlotInactive

This alert fires when a logical replication slot held by an Org Mover move on a GitLab.com Patroni primary has not been read for 15 minutes.

Org Mover moves an organization from one cell to another. The pilot is the service that coordinates a move. For each source database, a producer runs on the source cell, reads the organization’s changes from the database through siphon, a change-capture tool, and ships them to the target cell. The producer reads those changes through a replication slot on each source database, and a slot nobody reads keeps WAL for the whole database until it is read again, dropped or lost.

Contributing factors include:

  • An operator paused the move’s capture. This is expected, and the reason this alert does not page.
  • The move’s producer or siphon is down or keeps failing, or the producer halted on a data broker refusal.
  • PostgreSQL invalidated the slot at max_slot_wal_keep_size (wal_status reads lost). Nothing can read it any more, so the move cannot finish: abort it.
  • A move was aborted or finished, but its teardown has not dropped the slot yet. The teardown needs the Org Mover pilot, so a pilot that is down holds it up. The producer and siphon keep reading the slot without the pilot.

A slot also reads inactive for a few seconds while a siphon restarts, and while a move pauses its producers before tearing them down at its end. That pause is usually brief and the 15-minute wait lets it pass. But when the step that shuts down the producers at the end of a move keeps failing, they stay paused for as long as it fails, and abort is the only way out. So this alert can be a stuck move, not only a short pause.

Parts of the service affected:

  • pg_replication_slots_active == 0, for slot_type="logical" and slot_name=~"orgmover_slot_.*", on the primary only (pg_replication_is_replica == 0).
  • Source: postgres_exporter on the Patroni primaries, in the gitlab-gprd and gitlab-gstg Mimir tenants.
  • It does not check the WAL the slot keeps, so a lost slot, whose restart_lsn is empty, keeps firing.
  • Fires after 15 minutes, the same wait as PatroniClusterPaused, so a short pause or a producer restart can finish first. Clears as soon as the slot is read again or dropped. A lost slot keeps it firing until the move is aborted and its teardown drops the slot.
  • Silencing is right for a deliberate pause of a move’s capture: silence it for the planned window, matching the slot_name, and name the change request or the move in the silence. While it is silenced, OrgMoverReplicationSlotWALHigh still pages if the slot keeps too much WAL.
  • s3, no page. A paused or stalled move costs only WAL disk at first, and the WAL-size alert pages before that becomes a risk.

On the primary, list Org Mover’s slots, the WAL each keeps (retained) and the WAL its move has not read yet (unread):

SELECT slot_name, active, active_pid, wal_status,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS unread
FROM pg_replication_slots
WHERE slot_name LIKE 'orgmover_slot_%';

A slot keeps the WAL from the start of the oldest write transaction still open on the primary. When unread is small but retained is large, a long-running transaction holds the WAL, not the move: see postgres-long-running-transaction.

  1. Find the move. The slot name is orgmover_slot_ plus a hash of the move, its service and its database, so it does not say which organization is moving. A move holds one slot for each source database, so you can see more than one slot for the same move. Today, search the source cell’s producer logs for the slot name: when a move’s capture starts, the producer logs siphon producer created with the slot and the move’s id in transfer_id. In gstg, this query searches the gl-org-mover-gstg logging project for that line: add the slot name to it. If you cannot find it, ask in the #g_transport Slack channel. The capture row in the Org Mover GUI and the log line for a dropped slot will name each slot and its publication once https://gitlab.com/gitlab-org/cells/org-mover/org-mover/-/merge_requests/374 is merged.
  2. Paused on purpose: acknowledge, silence for the planned window as above, and check with whoever paused it that they plan to resume or abort.
  3. Not paused: resume, retry or abort. Check that the source cell is running, and whether the capture row shows the producer failing or halted. If the move is already aborted or finished and the slot is still there, check that the Org Mover pilot is running: the teardown that drops the slot needs it. Then:
    • The move should go on: resume its capture, or fix its producer. The slot then drains.
    • The producer halted because the data broker refused a captured change for storage: a stream is at its cap, or the broker’s store is full. Raise the full one, then take Retry on the capture row. See Broker storage. If the slot is already lost, Retry is refused and the move has to be aborted.
    • The move should stop: abort it. The abort’s teardown drops the slot and its publication.
    • The producer service on the source cell is down: start it first. An abort drops the slot through the producer, and waits until the producer answers.
    • The producer service is up: do not restart it while any move runs on it. A restarted producer forgets every move it holds, and a move that has not cut over yet can then only be aborted.
  4. Do not drop the slot of a live move by hand. If the move is still running, it does not notice the drop: the changes written on the source while the slot was gone never reach the target, and nothing reports it. Drop a slot by hand only when no move holds it any more.
  • No prior incidents recorded yet; this alert is new.
  • The Org Mover pilot and the source cell’s producer for the move.
  • The Patroni primary of the affected cluster, which owns the slot.
  • Primary contact: #g_transport Slack channel.