Skip to content

Siphon Logical Replication Slot (LRS) Resilience

Siphon consumes PostgreSQL WAL through a single logical replication slot per database cluster (prd_main_siphon_slot_1, prd_ci_siphon_slot_1, prd_sec_siphon_slot_1, and the stg_* equivalents). That slot is a single point of failure. If it is dropped, invalidated, or lost during a major PostgreSQL upgrade, Siphon can’t resume from where it was: every table of that database has a data gap. Recovery means recreating the slot and running consistency checks with corrections to repair the skipped rows, with a full re-snapshot into ClickHouse as the worst case.

To remove that single point of failure, the Patroni leader keeps one backup logical replication slot per cluster. A gated script drops and recreates it every hour, so a recent, usable fallback slot always exists. Siphon is configured with a replication.slot_priority_list. If the primary slot stops working, Siphon switches to the backup slot rather than failing and forcing a rebuild.

This runbook covers:

  • the design and the automation that drops and recreates the backup slot
  • how to verify that the automation is healthy, and which signal actually matters (catalog_xmin age, not only WAL)
  • what to do when the primary Siphon slot is lost, and how to get back to the canonical slot name without a data gap
  • the two retired designs and why each failed
  • the fallback path when all slots are lost (full rebuild)
  • the open gaps that still need closing

Owning teams: Database Operations owns the backup slot and its automation on Patroni. Platform Insights owns Siphon’s configuration and failover behaviour.

Origin: R&D Summit FY27, [Platform] Challenge #1: DIP everywhere.

Last reviewed: 2026-11-13.

When a Siphon slot is lost, replication can’t resume from the slot’s position. No data is lost in PostgreSQL, but there is a gap in every table of that database: a few thousand rows if the slot was gone for seconds, gigabytes if it was gone for minutes. The Siphon service runbook covers the recovery: recreate the slot, then run consistency checks with corrections to repair the skipped rows. That takes a while. A full re-snapshot of the database’s tables (1–2 weeks on production) is the worst-case fallback, and everything depending on ClickHouse is stale until it finishes.

A usable backup slot avoids the gap entirely: Siphon re-delivers at most one refresh interval of WAL, which ClickHouse absorbs as duplicates.

A WAL archive is not a fallback. A new logical slot always starts at the current WAL position, so PITR gives Siphon no resume point.

Measured from Mimir over the 14 days to 2026-11-07.

ClusterPGPrimary slotWAL generation (avg / hourly p95 / max hour)XID ratemax_slot_wal_keep_size
sec18prd_sec_siphon_slot_1~156 / ~221 / ~278 GB/h~0.9M/h1600 GiB
main17prd_main_siphon_slot_1~322 / ~433 / ~502 GB/h~10.9M/h3640 GiB
ci17prd_ci_siphon_slot_1~159 / ~222 / ~288 GB/h~18.9M/h6400 GiB

The registry cluster has only physical slots and no Siphon.

The XID rate matters as much as the WAL rate. It decides how far an unconsumed slot’s catalog_xmin falls behind between refreshes. ci uses XIDs about 20 times faster than sec.

Design: gated drop-and-recreate (“refresh”)

Section titled “Design: gated drop-and-recreate (“refresh”)”

There is one backup slot per cluster. siphon_slot_backup_refresh.sh drops and recreates it every hour. Each run:

  1. Takes a lock (gitlab-patroni!214 version only). An flock on LOCK_FILE makes an overlapping run exit cleanly. Hand-deployed copies don’t have it; see the note under Script.
  2. Checks for the leader. The check is SELECT NOT pg_is_in_recovery(). The script exits 0 on a standby. In the gitlab-patroni!214 version, a failed psql call exits 1 instead.
  3. Requires the primary Siphon slot to exist (gitlab-patroni!214 version). If it doesn’t, Siphon isn’t set up on this cluster or ENV/TYPE are wrong, so the run aborts with exit 1 before touching any slot.
  4. Recreates a missing backup slot straight away (gitlab-patroni!214 version). If the backup slot doesn’t exist, for example because an earlier run died between drop and create, or because the node was just promoted and its copy wasn’t sync-ready, the script creates it now, without the WAL gate, and stops. Creating a slot never touches the primary slot.
  5. Runs the health gate on the primary Siphon slot. The slot must be active, have wal_status = reserved, and be neither conflicting nor invalidated. The script then samples the slot’s retained WAL (pg_current_wal_lsn() − restart_lsn) every SAMPLE_EVERY seconds (10) for THRESHOLD_INTERVAL seconds (30). If the mean is above THRESHOLD_MB, it skips the run. A missing primary slot aborts with exit 1. Any other gate failure logs ABORT/SKIP and exits 0. In every case the existing backup slot is left untouched.
  6. Drops the backup slot. It refuses if the backup slot is active, meaning a consumer is attached, as happens after Siphon has failed over to it. The run then aborts without recreating.
  7. Waits DROP_RECREATE_DELAY seconds. Every current cron uses 60; the script default is 0. See Why the delay.
  8. Creates the new backup slot with pg_create_logical_replication_slot('<backup>', 'pgoutput', false, false, true), that is, failover = true. In the gitlab-patroni!214 version the create is bounded by statement_timeout = CREATE_TIMEOUT (300 s), and the script then checks that the slot exists with failover = t. A failure after the drop exits non-zero and logs backup slot ... is MISSING; the next run recreates it (step 4).
  9. Logs every step. This includes before and after snapshots of every slot’s catalog_xmin and age(datfrozenxid), plus a catalog_xmin line every 15 s during the wait (every 30 s in hand-deployed copies).

Why this works where the earlier designs did not

Section titled “Why this works where the earlier designs did not”
  • No decoding at all. The backup slot is never read, so the design does not depend on decode throughput. The advance-based design did, and it fell behind on high-WAL-rate clusters.
  • The horizon releases. Dropping the only unconsumed backup slot recomputes the cluster-wide slot horizon straight away (ReplicationSlotsComputeRequiredXmin, which the drop calls). Vacuum and the next slot creation both see the horizon fall back to the healthy primary slot’s position.
  • It is instant. No WAL is scanned, so it works at any WAL rate.
  • The primary slot is only read. It is never dropped, advanced or changed.

On the primary, the drop releases the horizon immediately. The delay is for the standbys:

  • The backup slot has failover = true, so each standby that syncs slots holds a copy of it.
  • Through hot_standby_feedback, the standby keeps reporting that copy’s old catalog_xmin back to the primary on its physical slot.
  • It keeps doing so until its slot-sync worker notices the drop (up to 30 s) and sends fresh feedback (wal_receiver_status_interval, 10 s by default).
  • A slot created within that window would inherit the old horizon through the standby’s physical slot.

The fixed 60 s covers this window. Polling until the physical slots’ catalog_xmin has caught up would be more robust; see Open gaps.

  • Retained WAL: about refresh interval × WAL rate. At hourly cadence that is roughly 150–280 GB on sec and ci, and up to ~500 GB on main. It is bounded by max_slot_wal_keep_size.
  • Catalog cleanup lag: about one interval of XIDs. Measured peaks just before a refresh are ~1.6M on sec and ~15–19M on ci. That is under 1% and about 9% of the 200M autovacuum_freeze_max_age trigger. A skipped refresh roughly doubles the peak, and consecutive skips keep adding to it.
  • Time with no backup slot: each run spends DROP_RECREATE_DELAY plus slot creation time without one. Creation waits for every transaction holding an XID at that moment to finish. Read-only queries don’t block it, but a long-running write transaction can block it for minutes.
  • Fallback window: if the primary slot is lost just before a refresh, Siphon has at most one interval of WAL to replay.
<env>_<db>_siphon_backup_slot_01
ClusterPrimary slotBackup slot
gprd secprd_sec_siphon_slot_1prd_sec_siphon_backup_slot_01
gprd ciprd_ci_siphon_slot_1prd_ci_siphon_backup_slot_01
gprd mainprd_main_siphon_slot_1prd_main_siphon_backup_slot_01
gstg sec / main / cistg_<db>_siphon_slot_1stg_<db>_siphon_backup_slot_01

The script only ever creates ..._backup_slot_01. If a cluster has more than one *_siphon_backup_slot_* slot, something else is creating them, and the cluster is in the INC-13574 failure mode. See Emergency: backup slots pinning the catalog horizon.

siphon_slot_backup_refresh.sh has so far been deployed by hand to /var/opt/gitlab/postgresql/siphon_slot_backup_refresh.sh, owned by gitlab-psql, mode 0750. gitlab-patroni!214 (in review) packages it as the gitlab-patroni::siphon_slot_backup_refresh recipe, and chef-repo!8338 adds that recipe to gstg-base-db-patroni-sec-v18 first. Once a role runs the recipe:

  • the script is at /usr/local/bin/siphon_slot_backup_refresh.sh and logs to /var/log/gitlab/postgresql/siphon_slot_backup_refresh.log, rotated daily (7 kept);
  • the cron is on every member of the cluster, and the leader check makes it a no-op on standbys, so it follows the leader after a failover;
  • settings come from role attributes under gitlab-patroni.siphon_slot_backup_refresh (env, type, threshold_interval, threshold_mb, sample_every, drop_recreate_delay, plugin, cron.minute/hour). The recipe passes them on the cron command line, not as crontab variables, so they don’t leak into other cron entries;
  • with enabled: false, or env/type unset, the recipe removes the cron, script and logrotate entry. Chef doesn’t undo a recipe that is taken out of the run list, so to take the automation off a cluster, set enabled: false and converge first, then remove the recipe;
  • Chef owns the crontab entry. A hand edit (for example commenting it out to pause it) is put back on the next chef run, within about 30 minutes. See Disable the automation on a cluster.
VariableRequiredMeaningValues in use
ENVyesFirst part of the slot nameprd / stg
TYPEyesSecond part of the slot namemain / ci / sec
THRESHOLD_MByesMean retained WAL on the primary slot above which the run is skipped1500 (gstg sec, chef-repo!8338), 3000 (gprd sec test), 3500 (gprd ci test); Chef default 2000
THRESHOLD_INTERVALyesSeconds over which retained WAL is sampled30
SAMPLE_EVERYnoSeconds between samples10 (default)
DROP_RECREATE_DELAYnoSeconds between drop and recreate60 (Chef default and every cron so far; script default 0)
CREATE_TIMEOUTnostatement_timeout for the slot create, in seconds (gitlab-patroni!214 only)300 (default; the recipe doesn’t pass it yet)
PLUGINnoOutput pluginpgoutput (default)
LOG_FILEnoLog pathChef: /var/log/gitlab/postgresql/siphon_slot_backup_refresh.log; script default /tmp/siphon_slot_backup_refresh.log
LOCK_FILEnoflock path (gitlab-patroni!214 only)The script file itself (default), opened read-only, so any user that can read the script can take the lock

Hourly at minute 0. With Chef, the entry is on every member and looks like this (gstg sec, chef-repo!8338):

# Chef Name: siphon_slot_backup_refresh
0 * * * * ENV=stg TYPE=sec THRESHOLD_INTERVAL=30 THRESHOLD_MB=1500 SAMPLE_EVERY=10 DROP_RECREATE_DELAY=60 PLUGIN=pgoutput LOG_FILE=/var/log/gitlab/postgresql/siphon_slot_backup_refresh.log /usr/local/bin/siphon_slot_backup_refresh.sh > /dev/null 2>> /var/log/gitlab/postgresql/siphon_slot_backup_refresh.log

The hand-deployed entries below, as gitlab-psql on the leader only, were used in the gprd validation tests. Both tests have ended (see Current status):

# Siphon slot backup refresh - gprd sec (production#23400)
0 * * * * ENV=prd TYPE=sec THRESHOLD_INTERVAL=30 THRESHOLD_MB=3000 DROP_RECREATE_DELAY=60 /var/opt/gitlab/postgresql/siphon_slot_backup_refresh.sh >> /tmp/siphon_slot_backup_refresh.log 2>&1
# Siphon slot backup refresh - gprd ci (production#23426)
0 * * * * ENV=prd TYPE=ci THRESHOLD_INTERVAL=30 THRESHOLD_MB=3500 DROP_RECREATE_DELAY=60 /var/opt/gitlab/postgresql/siphon_slot_backup_refresh.sh >> /tmp/siphon_slot_backup_refresh.log 2>&1

The script already tees to LOG_FILE, so with this redirect each line probably shows up twice in the log. The Chef cron in gitlab-patroni!214 sends stdout to /dev/null instead.

Interval: hourly. Platform Insights confirmed that 1h and 2h both work for Siphon, and 1h was chosen. A shorter interval lowers the catalog_xmin peak between refreshes and gives a shorter fallback window. A longer one does the reverse.

This is set in the ArgoCD applications repo (gitlab-com/gl-infra/argocd/apps), services/siphon/env/orbit-<env>/values.yaml, under each producer deployment:

deployments:
postgres-sec-producer:
appConfig:
replication:
publication_name: stg_sec_siphon_publication_1
slot_name: stg_sec_siphon_slot_1
slot_priority_list:
- stg_sec_siphon_backup_slot_01

ArgoCD auto-sync is off for Siphon, so a values change needs a manual sync.

How Siphon chooses a slot (SetupReplicationSlot):

  • It walks [slot_name] + slot_priority_list in order. A slot is skipped if it is missing, has wal_status = 'lost', has conflicting = true, or has invalidation_reason set. Siphon connects to the first slot that works and resumes from that slot’s confirmed_flush_lsn.
  • Selection runs when the producer starts. A dropped or invalidated slot makes replication fail, and the producer then restarts in-process (no pod restart) and selects again. That is how failover happens.
  • A plain reconnect keeps the slot already chosen. Once Siphon is on the backup slot it stays there, even after the primary slot is recreated. Switching back needs a deliberate restart (see Recovery).
  • If no slot works, the producer logs no functional replication slot found among [...] and keeps restarting.
EnvironmentRefresh script + cronSiphon slot_priority_listNotes
gstg sec⚠️ Not refreshing since the 2026-11-04 failover test (production-engineering#30302). The backup slot’s catalog_xmin age was ~1.4M and still climbing on 2026-11-12✅Failover validated end to end (2026-11-04). First Chef rollout target: chef-repo!8338 (THRESHOLD_MB=1500)
gstg main⚠️ Backup slot missing since a leader change on 2026-10-27✅This is the failover-orphan gap
gstg ci⚠️ Backup slot missing since a leader change on 2026-10-27✅Same
gprd sec✅ Validated over a week (2026-11-03 to 11-10), then removed. The backup slot’s metrics end at ~2026-11-10 19:00 UTC❌production#23400, THRESHOLD_MB=3000, leader patroni-sec-v18-101. No backup slot until the Chef rollout
gprd ci✅ Validated, then removed (2026-11-06 to 11-08)❌production#23426, THRESHOLD_MB=3500, leader patroni-ci-v17-27. No backup slot until the Chef rollout
gprd main❌ Not scheduled❌Highest WAL rate; needs its own threshold measurement
Chef packaging🟡 In review: gitlab-patroni!214 (recipe, cookbook 1.17.0) and chef-repo!8338 (gstg sec role)n/aCheck the gitlab-patroni pin before the role change (see Not yet packaged in Chef). Then gstg ci and main, which also fixes the gstg rows above, then gprd
Patroni ignore_slots✅ gstg sec, ci, main v18 · ✅ gprd sec-v18, ci-v17, main-v17n/agprd done 2026-11-16 in production#23458 (~C2, chef-repo!8309). Run the old-leader cleanup after every leader change. See Patroni and the Siphon slots
Backup-slot lag alert exclusion🟡 runbooks!12101 (open)n/aSee False-positive lag alerts on the backup slot

gprd sec refresh, 2026-10-14 21:00 to 10-15 ~15:00 UTC (production#23006, THRESHOLD_MB=1500):

  • 18 recreates in 18 hours, with no skipped runs.
  • The backup slot’s catalog_xmin age dropped to ~35–80K right after each recreate and climbed to ~0.6–1.6M by the next run. The peak over the whole window was ~1.6M, under 1% of the 200M anti-wraparound trigger.
  • The primary slot prd_sec_siphon_slot_1 was unaffected throughout (~2.5K–25K).
  • Teardown was clean. After the test the backup slot was dropped and the horizon went back to baseline (~13K).
  • datfrozenxid stayed at ~10% of the wraparound budget throughout.

Separate testing showed the gate skipping refreshes during the recurring daily primary-slot spike and resuming afterwards.

gprd ci refresh, 2026-11-06 21:05 to 11-08 ~19:30 UTC (production#23426, THRESHOLD_MB=3500):

  • 46 refreshes from 11-06 22:02 to 11-08 19:02 UTC, one per hour, with no skipped runs. The window was planned as 24 hours and ran about 46.
  • The backup slot’s catalog_xmin age dropped to ~0.1–0.7M after each refresh and reached ~7–19M before the next. That is about 10 times sec’s sawtooth, because ci burns ~19M XIDs/h against ~1M/h on sec, and still under 10% of the 200M anti-wraparound trigger.
  • The primary slot prd_ci_siphon_slot_1 stayed healthy throughout (catalog_xmin age ≤ 0.31M).
  • pg_xid_wraparound saturation on ci stayed at 9–13%, the same as the week before.
  • Peak backup-slot lag was 153 GiB, 2.4% of ci’s max_slot_wal_keep_size (6.25 TiB).
  • Teardown was clean: the backup slot was dropped and the cron removed.
  • SiphonLogicalReplicationSlotLagHigh fired on the backup slot again on 11-08, after the CR’s 26h silence had expired. See Open gaps.

gprd sec refresh, one week, 2026-11-03 22:05 to 11-10 ~19:00 UTC (production#23400, THRESHOLD_MB=3000; results and graphs in gitlab-org/gitlab#605537):

  • The hourly sawtooth held for the whole week: the backup slot’s catalog_xmin age dropped to ~60–250K after each refresh and was normally ~0.3–1.6M before the next one.
  • The gate skipped refreshes only while the primary slot was behind, during the daily spikes on 11-06 (~15:30–17:00 UTC) and 11-09 (~00:30–02:00 UTC), and resumed on its own. The highest point of the week, ~2.9M on 11-06, came from those skips. That is under 2% of the 200M anti-wraparound trigger.
  • The primary slot prd_sec_siphon_slot_1 stayed at ~1–25K outside those spikes (spike peaks ~0.5M and ~0.8M).
  • pg_xid_wraparound saturation on sec stayed at 9.1–9.4% all week.
  • The backup slot was dropped at the end of the test: its series in Mimir end at about 11-10 19:00 UTC, and only prd_sec_siphon_slot_1 remains on the four members.
  • With these results, the gated drop-and-recreate refresh is the agreed design (decision).

Siphon failover, gstg sec, 2026-11-04 (image siphon:0.0.156-beta, postgres-sec-producer; record):

Time (UTC)Event
10:21:29Terminated the walsender and dropped stg_sec_siphon_slot_1. Last acked LSN B80B/EA000060
10:21:31replication slot not functional, trying next, then using replication slot stg_sec_siphon_backup_slot_01, resuming from B80B/D426AD50
10:21:41Past the old stop point; caught up in under 10 s
10:23:51Recreated stg_sec_siphon_slot_1
10:24:01Backup slot past the new slot’s start LSN, so safe to restart
10:25:11After a pod restart, using replication slot stg_sec_siphon_slot_1

Failover took ~1.6 s. The backup slot was ~350 MiB of WAL behind, and Siphon replayed that in under 10 s. Most of that WAL was likely for tables outside the publication, which pgoutput skips. Switching back replayed ~16 MiB of deliberate overlap. In both directions the resume LSN was behind the last acked LSN, so nothing was skipped. ClickHouse’s ReplacingMergeTree absorbs the duplicates.

An earlier failover test on 2026-08-27 showed the same behaviour under the retired multi-slot configuration.

WAL retention is the obvious cost of a slot, but it is not the dangerous one, because max_slot_wal_keep_size bounds it. Nothing bounds the horizon directly, and a pinned horizon is what caused INC-13574. Check it first, across all slots, physical included:

SELECT slot_name, slot_type, active, synced, temporary,
age(catalog_xmin) AS catalog_xmin_age,
age(xmin) AS xmin_age,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal,
wal_status
FROM pg_replication_slots
ORDER BY greatest(age(catalog_xmin), age(xmin)) DESC NULLS LAST;

A healthy leader, partway through a run, shows:

  • exactly one *_siphon_backup_slot_01 row, with active = f;
  • the backup slot’s catalog_xmin_age moving in a sawtooth that never climbs steadily, within the ranges below;
  • the primary slot in the range below;
  • physical slots (standbys) low as well, since they carry standby feedback;
  • wal_status = reserved or extended on both Siphon slots. unreserved means the slot loses WAL at the next checkpoint, so act now.
ClusterBackup slot, just after a refreshBackup slot, just before the nextPrimary slot
sec~10–80K~0.8–1.6Mlow tens of thousands
ci~0.2–0.4M~15–19M~0.1–0.3M
mainTBDTBDnormally ~0.1–0.3M, with periodic spikes

Escalate if catalog_xmin_age on any slot keeps climbing without dropping back, or if age(datfrozenxid) is rising while pg_txid_xmin_age stays flat. That divergence is the INC-13574 signature.

Metrics: pg_replication_slots_catalog_xmin_age, pg_replication_slots_xmin_age, pg_txid_xmin_age, pg_database_wraparound_age_datfrozenxid. wal_status is not exported, so check it with SQL.

On standbys: is the backup slot sync-ready?

Section titled “On standbys: is the backup slot sync-ready?”

failover = true only marks the slot for synchronization (PG17+). It reaches a standby only if that standby has sync_replication_slots = on, hot_standby_feedback = on, primary_slot_name set, and dbname in primary_conninfo, and streams directly from the primary. Cascading standbys cannot sync. After a failover, the slot is usable only if, before the failover, it showed synced = t and temporary = f on the standby that was promoted. A slot that is never consumed may stay temporary = t on a standby, so check:

-- on each replica
SELECT slot_name, synced, temporary, invalidation_reason
FROM pg_replication_slots WHERE slot_name LIKE '%siphon%';

Our clusters use PostgreSQL’s native slot sync (sync_replication_slots = on in the Patroni roles), not Patroni’s permanent slots. Patroni is told to leave the Siphon slots alone with ignore_slots (see Patroni and the Siphon slots).

PostgreSQL replication slot xmin age was built for this failure mode after INC-13566 (runbooks!11389). It puts the whole chain of signals on one page: slot catalog_xmin age, cluster xmin age, datfrozenxid age, autovacuum worker saturation and analyze rate.

Also useful:

Terminal window
sudo gitlab-patronictl list # confirm this node is the leader
sudo grep -r siphon_slot_backup_refresh /var/spool/cron/crontabs/ # whichever user owns it
sudo tail -50 /var/log/gitlab/postgresql/siphon_slot_backup_refresh.log # Chef
sudo tail -50 /tmp/siphon_slot_backup_refresh.log # hand-deployed

A healthy log on the leader shows === Done === once an hour, or an explicit reason when a run does nothing:

  • SKIP: mean WAL ... > threshold: normal during the daily spike;
  • ABORT: primary slot ... does not exist / is NOT active / wal_status=... / conflicting/invalidated: the primary slot is unhealthy. Check Siphon;
  • Not the leader node: normal on standbys, where the Chef cron also runs;
  • Another refresh run holds ...: runs overlapped (gitlab-patroni!214 version only);
  • Backup slot ... is MISSING - recreating it now: the previous run failed after the drop, or the node was just promoted without a sync-ready copy. One occurrence is fine;
  • FAILED (exit N): backup slot ... is MISSING: the create failed or timed out (CREATE_TIMEOUT, usually a long-running write transaction). The cluster has no backup slot until the next run. If it repeats, check for long transactions and create the slot by hand.

Consecutive skips or aborts over many hours are not normal. They mean the backup slot is ageing with nothing resetting it.

The examples below use gprd sec. Substitute <env>_<db> for other clusters.

Symptoms: the Siphon producer restarts, and the primary slot is either missing from pg_replication_slots or present with wal_status = 'lost', conflicting = true, or invalidation_reason set.

What happens automatically (only where slot_priority_list is configured, currently staging only): the producer restarts in-process, walks the slot list and connects to the backup slot within seconds. It replays at most one refresh interval of WAL. No re-snapshot is required.

Step 1: Confirm which slot Siphon is using

Section titled “Step 1: Confirm which slot Siphon is using”
SELECT slot_name, active, active_pid, wal_status,
age(catalog_xmin) AS catalog_xmin_age,
pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS lag
FROM pg_replication_slots
WHERE slot_name ILIKE '%siphon%'
ORDER BY active DESC, slot_name;

The row with active = t and a live active_pid is the one in use. Confirm it in the producer logs:

Terminal window
glsh kube use-cluster orbit-prd --no-proxy # orbit-stg for staging
kubectl logs -n siphon -l app=postgres-<db>-producer --tail=200 | grep -E "replication slot"

The script should not touch the backup slot while Siphon is using it. It aborts when the primary slot is missing or inactive, and it refuses to drop an active backup slot. Pause it anyway, so that no refresh runs while you recreate and switch slots.

  • Hand-deployed: as gitlab-psql on the leader, crontab -e and comment out the siphon_slot_backup_refresh line.

  • Chef-managed: a crontab edit is put back by the next chef run. On every member (the cron runs wherever the leader is), disable chef with a reason first, then comment out the line:

    Terminal window
    sudo chef-client-disable "Siphon slot recovery, <issue link>"
    sudo crontab -e -u <cron user> # comment out the siphon_slot_backup_refresh line

    Re-enable chef in Step 7. For a longer pause, set enabled: false in the role instead (see Disable the automation on a cluster).

Confirm that lag on the slot now in use is falling. If it isn’t, treat this as a high replication lag incident first and come back to the restore afterwards.

SET statement_timeout = '10min'; -- creation waits for in-progress write transactions
SELECT * FROM pg_create_logical_replication_slot(
'prd_sec_siphon_slot_1', -- slot name
'pgoutput', -- plugin
false, -- temporary
false, -- twophase
true -- failover
);

A cancelled create leaves no slot behind, so retrying is safe. Keep failover = true: it marks the slot for sync to the standbys (see sync-ready).

Step 5: Wait for overlap before switching back

Section titled “Step 5: Wait for overlap before switching back”

Wait until the backup slot that Siphon is consuming has reached the new slot’s confirmed_flush_lsn:

SELECT s1.slot_name AS target_slot,
s1.confirmed_flush_lsn AS target_lsn,
s2.slot_name AS backup_slot,
s2.confirmed_flush_lsn AS backup_lsn,
(s1.confirmed_flush_lsn <= s2.confirmed_flush_lsn) AS safe_to_switch,
pg_wal_lsn_diff(s2.confirmed_flush_lsn, s1.confirmed_flush_lsn) AS overlap_bytes
FROM pg_replication_slots s1
CROSS JOIN pg_replication_slots s2
WHERE s1.slot_name = 'prd_sec_siphon_slot_1'
AND s2.slot_name = 'prd_sec_siphon_backup_slot_01';

Proceed only when safe_to_switch = t, which usually takes a few minutes. overlap_bytes is the amount Siphon will replay. That is harmless, because ReplacingMergeTree deduplicates it downstream. This check assumes Siphon acknowledges an LSN only after it has durably published everything before it.

Step 6: Restart Siphon onto the canonical slot

Section titled “Step 6: Restart Siphon onto the canonical slot”

Siphon stays on the backup slot until it is restarted:

Terminal window
kubectl rollout restart deployment/postgres-<db>-producer -n siphon
kubectl logs -n siphon -l app=postgres-<db>-producer --tail=100 \
| grep -E "using replication slot|resuming from replication slot LSN"

Confirm that it picked the canonical slot with no not functional warning.

Uncomment the cron entry, or on Chef-managed nodes run sudo chef-client-enable and sudo chef-client, which restores it. Then run the script once by hand (see Manual operations). From the log and from pg_replication_slots, confirm that the backup slot was recreated and that its catalog_xmin_age is back near the baseline for the cluster.

On a Chef-managed node, copy the exact command from the crontab entry (sudo grep -r siphon_slot_backup_refresh /var/spool/cron/crontabs/) and run it as the user that owns that crontab, so the settings match the role. Hand-deployed:

Terminal window
sudo su - gitlab-psql
ENV=prd TYPE=sec THRESHOLD_INTERVAL=30 THRESHOLD_MB=3000 DROP_RECREATE_DELAY=60 \
/var/opt/gitlab/postgresql/siphon_slot_backup_refresh.sh

Don’t run it within a few minutes of the top of the hour. Hand-deployed copies have no overlap guard, so a manual run and the cron run could both drop and recreate the slot at once. The gitlab-patroni!214 version takes an flock on the script file itself, opened read-only, so the second run exits cleanly whichever user started the first.

SELECT * FROM pg_create_logical_replication_slot(
'prd_sec_siphon_backup_slot_01', 'pgoutput', false, false, true);

This is safe only when the slot is not in use. If Siphon has failed over, the backup slot is the one being consumed, and dropping it forces a full re-snapshot. Check first:

SELECT slot_name, active, active_pid
FROM pg_replication_slots WHERE slot_name = 'prd_sec_siphon_backup_slot_01';
-- only if active = f:
SELECT pg_drop_replication_slot('prd_sec_siphon_backup_slot_01');

Drop it on the primary. On a standby, synced slots can’t be dropped by hand; the slot-sync workers remove their copies.

After a switchover or failover: clean up the old leader

Section titled “After a switchover or failover: clean up the old leader”

Do this after every leader change on a cluster with Siphon slots: planned switchovers, manual failovers, and unplanned failovers (including a watchdog reset).

Why. The old leader created the Siphon slots while it was the primary, so its copies have synced = f. After it is demoted:

  • PostgreSQL’s slot-sync worker stops at a non-synced slot with the same name: exiting from slot synchronization because same name slot "<name>" already exists on the standby. This is the same on PG17 and PG18 (PG17 slotsync.c, PG18 slotsync.c). The node has no sync-ready copy, so a later promotion of it brings back a stale Siphon slot.
  • The stale slots never advance. They retain WAL on the old leader, and with hot_standby_feedback = on the standby reports their catalog_xmin to the new primary, which can hold back the new primary’s catalog horizon.
  • Patroni no longer drops them, because of ignore_slots.

A node that was itself promoted from a standby can already have synced = t copies (gstg main-103 on 2026-11-04). Then there is nothing to clean up.

The same applies to a replica built from a snapshot of the leader’s disk. The snapshot carries the leader’s own synced = f Siphon slots, so the new replica’s slot sync stops at them, exactly as on a demoted leader. Before ignore_slots, Patroni dropped them when it started on the new node; with ignore_slots, nothing does. Prefer snapshots taken from a replica, whose copies have synced = t. If the source was the leader, run the guarded drop below on the new replica once it is streaming.

Steps:

  1. On the new leader first: check that the primary Siphon slot is there.

    Terminal window
    sudo gitlab-psql -AtXc "select slot_name, active, failover, confirmed_flush_lsn from pg_replication_slots where slot_name like '%_siphon_slot_1'"

    If *_siphon_slot_1 is missing, the promoted node had no sync-ready copy. Stop here and don’t drop anything. Go to Recovery: the primary Siphon slot is lost. The old leader’s stale copy is then the only record of Siphon’s last position before the leader change, and the LSN-overlap check needs it. Keep it until Recovery is done.

  2. On the old leader, once it is streaming as a replica: confirm the error, and list what the drop would remove with its LSNs. Paste the output into the CR or incident, so the positions are kept after the drop:

    Terminal window
    sudo grep "exiting from slot synchronization because same name slot" /var/log/gitlab/postgresql/*.csv | tail -2
    sudo gitlab-psql -AtXc "select slot_name, synced, active, restart_lsn, confirmed_flush_lsn from pg_replication_slots where slot_type = 'logical' and plugin = 'pgoutput' and database = 'gitlabhq_production' and not synced and not active and pg_is_in_recovery()"
  3. Drop them. The guard only acts on a standby (pg_is_in_recovery()), and only on non-synced, inactive pgoutput slots, so on the primary it returns no rows:

    Terminal window
    sudo gitlab-psql -AtXc "select slot_name, pg_drop_replication_slot(slot_name) from pg_replication_slots where slot_type = 'logical' and plugin = 'pgoutput' and database = 'gitlabhq_production' and not synced and not active and pg_is_in_recovery()"
  4. Success: within about a minute the slot-sync worker recreates the Siphon slots on that node with synced = t and temporary = f, and no new exiting from slot synchronization lines appear. On the new primary, the old leader’s physical slot shows a catalog_xmin age in the same range as the other replicas. A never-consumed backup slot can stay temporary = t for a while (see On standbys).

  5. On the new leader: check that the Siphon consumer has reconnected (active = t on *_siphon_slot_1), then follow Patroni failover orphans the backup slot for the backup slot and the refresh cron.

This is the same guarded drop used in production#23172 (step 3c) and production#23352 (step 7), where it removed exactly the stale Siphon slots on gstg sec and ci.

  1. Stop the cron:
    • Chef-managed: set "enabled": false under gitlab-patroni.siphon_slot_backup_refresh in the cluster’s role, merge, and run sudo chef-client on every member. The recipe removes the cron, the script and the logrotate entry. Only then remove the recipe from the run list, if wanted; removing it first leaves the cron behind.
    • Hand-deployed: comment out or remove the cron entry, as gitlab-psql.
  2. Drop the backup slot on the leader. Check active = f first, as above. Disabling the recipe does not drop the slot.
  3. Confirm that catalog_xmin age is back to baseline. On the primary it happens within seconds. Standbys’ physical slots may report the old value for another ~30–40 s.
  4. Expire any Alertmanager silence that was created for the slot.

Emergency: backup slots pinning the catalog horizon

Section titled “Emergency: backup slots pinning the catalog horizon”

Signature:

  • age(datfrozenxid) is rising while pg_txid_xmin_age stays flat;
  • one or more *_siphon_backup_slot_* slots have a high catalog_xmin_age that only climbs;
  • autovacuum workers are saturated on wraparound-prevention vacuums;
  • dead tuples are piling up, and the analyze rate is zero.

Act immediately: this escalates to user-visible impact.

  1. Stop whatever is creating slots: comment out the cron entry.

  2. Drop every unconsumed backup slot. The horizon only releases once every slot older than the healthy primary slot is gone.

    SELECT slot_name, active, age(catalog_xmin)
    FROM pg_replication_slots
    WHERE slot_name LIKE '%siphon_backup_slot%';
    SELECT pg_drop_replication_slot('<each slot with active = f>');
  3. Confirm that catalog_xmin age drops to baseline; this is immediate on the primary. age(datfrozenxid) follows once autovacuum has finished its anti-wraparound passes over the catalogs.

  4. Autovacuum resumes on its own, but statistics are slow to catch up. Run a manual ANALYZE on the busy tables that were starved. In INC-13566 it was the stale statistics, not the dead tuples themselves, that flipped query plans and caused the 500s.

  5. Escalate to #g_database_operations.

Retired approaches (read before changing anything)

Section titled “Retired approaches (read before changing anything)”

Approach 1: three hourly-rotated backup slots ❌

Section titled “Approach 1: three hourly-rotated backup slots ❌”

What it was (Aug 2026): several backup slots per cluster. A new one was created every hour, and the stale one dropped a few hours later, so Siphon always had a slot at most a few hours old. It ran on staging from 2026-08-10 and was enabled on gprd sec.

What happened: INC-13566 (Sev2, production#22807) and INC-13574 (Sev4, production#22817), gprd sec, 2026-08-25 to 08-27, with 3h 52m of user impact. The combined review is production#22812 (confidential).

  • From about 2026-08-25 13:45 UTC, age(datfrozenxid) on patroni-sec-v18 rose from its ~200M baseline to ~266M by 08-27, in lockstep across all three databases. pg_txid_xmin_age stayed at 505 XIDs, while the four backup slots showed a catalog_xmin age of about 265.9M.
  • Autovacuum stopped making progress across the instance. Dead tuples piled up on the largest, busiest tables, and some had not been vacuumed or analyzed for days.
  • Stale statistics flipped the pipeline security tab queries to parallel sequential scans. CPU saturated, and there were 500s from 08:20 to 11:51 UTC on 08-27.
  • Each backup slot also held 150–230 GB of WAL during its 3-hour life, about 600 GB pinned at once.
  • Recovery: a manual ANALYZE on the busy tables that had been starved, and dropping all four backup slots. The horizon released immediately, age(datfrozenxid) returned to baseline once autovacuum caught up, and autovacuum resumed.
  • gprd main showed the same climbing pattern, but it had enough headroom that it never became user-visible.

Why rotation cannot work (checked against the PostgreSQL source):

  1. A logical slot’s catalog_xmin moves only when WAL is decoded through the slot and a confirmed position is recorded (LogicalConfirmReceivedLocation). A slot nobody reads never advances, however young it is.
  2. A new logical slot is born already old. pg_create_logical_replication_slot() sets the starting catalog_xmin to GetOldestSafeDecodingTransactionId(catalogOnly = true). That is the oldest of the next XID, any running transaction’s XID, and the xmin / catalog_xmin of every existing slot, physical slots included. So every hourly slot inherited the frozen horizon of the oldest unconsumed slot still alive. Rotation moved the floor forward only by the gap between consecutive creations, while XIDs advanced at production rate.
  3. What catalog_xmin actually pins. A logical slot’s catalog_xmin holds back cleanup of system catalogs (including shared catalogs) and user_catalog_table tables only, not ordinary tables. This has been true since logical decoding arrived in 9.4. Since PG14 it is implemented in ComputeXidHorizons. The chain from there to user impact:
    • The slot horizon is a single cluster-wide value, applied to the catalogs of every database whichever database the slot belongs to. So no catalog anywhere could have its relfrozenxid advanced past it.
    • datfrozenxid is capped by the same horizon (vac_update_datfrozenxid), so all three databases aged in lockstep.
    • Once they passed autovacuum_freeze_max_age (200M), every catalog qualified for an anti-wraparound vacuum on every pass, and none could finish it.
    • While any database is past that limit, the autovacuum launcher sends each new worker to the database with the oldest datfrozenxid and ignores the rest (do_start_worker).
    • Ordinary tables were never blocked. Their dead tuples grew because no worker got to them, so stale statistics followed, then plan flips, then CPU saturation.
  4. Two symptoms, one cause. The same unconsumed slots pin WAL retention (restart_lsn) and the horizon (catalog_xmin). max_slot_wal_keep_size looks only at restart_lsn. A slot that inherited an old catalog_xmin still has a young restart_lsn, so the WAL limit never triggers and the horizon has no bound.

Decision: retired, and must not be re-enabled anywhere, including Dedicated and Self-Managed once Siphon ships there. The mechanism is pure PostgreSQL and would happen the same way.

The fallback concept survived: the 2026-08-27 staging test proved that Siphon can fail over to a backup slot without data loss. What had to change was how the backup slot is kept current. It must be consumed, advanced, or recreated from a clean floor, never just rotated alongside the slots before it.

Approach 2: permanent slots advanced with pg_replication_slot_advance ❌

Section titled “Approach 2: permanent slots advanced with pg_replication_slot_advance ❌”

What it was (Sep 2026): permanent slots advanced every hour to fixed WAL-byte offsets behind the primary slot’s confirmed_flush_lsn. It was stateless, only ever moved forward, and never dropped a slot. It was validated on all three staging clusters.

Why it failed on gprd sec: the production#22859 window ran on 2026-09-11. The controlled tests in production#22939 ran on 2026-09-15.

pg_replication_slot_advance moves confirmed_flush_lsn to the requested target. But restart_lsn, which catalog_xmin is tied to, can only advance to a restart point recorded at a RUNNING_XACTS record.

Test 1Test 2
Bytes scanned (restart_lsn delta)515 MB4,041 MB
Elapsed20.6 s2 m 30 s
restart_behind_confirmed after4.2 GB30 GB
catalog_xmin ageroserose

The gap grew instead of closing.

  • Throughput view (RCA): restart_lsn advanced at ~1.6 GB/min against sec’s ~2.5 GB/min of WAL, so it lost ground every run.
  • Mechanism (source review): an advance call decodes up to the target and confirms only once, at the end.
    • While decoding, the slot keeps only the first candidate restart_lsn / catalog_xmin past its previous confirmed_flush_lsn, and discards the rest (LogicalIncreaseRestartDecodingForSlot / LogicalIncreaseXminForSlot).
    • So after each call, restart_lsn lands just past the previous confirmed position, however far the target moved.
    • Test 2 decoded ~34 GB in 2m 30s, about six times sec’s WAL rate. Decode throughput was not the binding limit, yet restart_lsn moved only about 4 GB.

Either way, keeping the horizon tight would need a tight loop of small advances, each re-decoding the WAL behind it: a home-made consumer with worse semantics than a real one.

Staging passed because its WAL rate is 50–75× lower, so it could not hit this limit. The lesson for testing: validate anything that scales with volume against production-like WAL rates.

Impact: none. datfrozenxid stayed at 10.0% of the wraparound budget throughout. Both tests were time-boxed, with an abort plan agreed in advance. Rollback was clean: dropping the slot took catalog_xmin from ~1.30M back to ~19K within seconds.

Approach considered but not built: a real second consumer

Section titled “Approach considered but not built: a real second consumer”

Siphon would run a second consumer against the backup slot, deliberately trailing the primary by a set lag. A streaming consumer acknowledges continuously, so catalog_xmin and restart_lsn would advance continuously, and the failure mode would be designed out.

This was ruled out for now on cost and scope. It doubles the consumers per cluster (a second full logical decode of the WAL stream), and the lag has to be enforced through the consumer’s feedback, which is Siphon development work. Platform Insights judged it feasible but said it would need changes to the null-producer implementation and a lag mechanism based on WAL volume. It stays an option if the refresh design proves insufficient.

This has already happened on gstg: the main and ci backup slots disappeared after leader changes on 2026-10-27 and have not come back.

The backup slot has failover = true, so it can survive a leader change if it was sync-ready on the promoted standby. The manually deployed cron does not move with the leader. After a failover:

  • the old leader’s script exits at its leader check and stops refreshing;
  • nothing refreshes the slot on the new leader, so it holds WAL and catalog_xmin there indefinitely. This is the INC-13574 failure mode with one slot instead of four;
  • any Alertmanager silence pinned to the old fqdn no longer matches, so the Siphon lag alert pages for the new node.

On a hand-deployed cluster, a Patroni failover needs manual action (from production#23426):

  1. On the old leader, as gitlab-psql, comment out the cron entry.
  2. On the old leader, run the guarded drop in After a switchover or failover.
  3. Find the new leader: sudo gitlab-patronictl list.
  4. On the new leader, check that the backup slot has active = f, then drop it.
  5. Verify that WAL retention and catalog_xmin recover on the new leader.
  6. Expire the Alertmanager silence, or tell the SRE on-call if the alert has already paged for the new fqdn.

Then redeploy the script and cron on the new leader, or leave the cluster without a backup slot until you do.

The permanent fix is gitlab-patroni!214. It installs the script and the same cron on every node, and the leader check turns the standbys into no-ops. After a leader change, the new leader’s next run (within the hour) either refreshes the synced copy of the backup slot, or, if that copy wasn’t sync-ready and the slot is missing, recreates it. What still needs a person on a Chef-managed cluster:

The related corrective action is production-engineering#29661, “Prevent orphaned siphon backup replication slots from pinning vacuum” (Verification, due 2026-11-23).

Patroni 3.3.4 drops any replication slot that is not in its slots: config or ignore_slots, on primaries and replicas, whenever it re-reads the slot list: when Patroni starts, after Postgres restarts (including a demotion), and when a paused cluster is resumed. That is how gstg sec lost both Siphon slots during a switchover on 2026-10-13 (production#23172).

  • gstg: fixed. The sec, ci and main v18 roles set bootstrap.dcs.ignore_slots to {type: logical, plugin: pgoutput, database: gitlabhq_production} (chef-repo!8244). It has to be in the role: the cookbook rewrites the DCS config from bootstrap.dcs on every chef run, so a patronictl edit-config alone is reverted.
  • gprd: fixed on 2026-11-16. production#23458 (~C2) with chef-repo!8309 (merged 2026-11-16 09:09 UTC) added the same matcher to gprd-base-db-patroni-sec-v18, gprd-base-db-patroni-ci-v17 (its c4 sub-roles inherit it) and gprd-base-db-patroni-main-v17, and it is in DCS on all three clusters. The Siphon slots prd_sec_siphon_slot_1, prd_ci_siphon_slot_1 and prd_main_siphon_slot_1 are no longer dropped by a Patroni restart, switchover, pause/resume or failover. The flip side: on these clusters too, Patroni no longer cleans up the old leader’s stale copies, so the old-leader cleanup is needed after every gprd leader change.

What we’ve learned from rolling it out:

  • One matcher covers every Siphon slot. Patroni matches exact values, so {type: logical, plugin: pgoutput, database: gitlabhq_production} protects the primary slot, the backup slot and any future pgoutput slot in that database, with no list of slot names to keep up to date.
  • Only the leader’s chef run applies it. The cookbook runs patronictl edit-config --force --replace from bootstrap.dcs on the leader. A replica’s chef run doesn’t touch DCS. The setting is dynamic: Patroni applies it on its next HA loop, with no restart or switchover.
  • Verify it with sudo gitlab-patronictl show-config | grep -A4 "^ignore_slots" on the leader, and check that no member logs Dropped unknown replication slot or Exception when changing replication slots in /var/log/gitlab/patroni/patroni.log afterwards.
  • Trade-off: Patroni no longer cleans up orphaned pgoutput slots in gitlabhq_production. A retired Siphon slot has to be dropped by its owner, and stale synced = f copies need the guarded drop after every leader change and after building a replica from a leader snapshot.
  • New roles need it too. When ci and main move to v18 roles, those roles need the same matcher. Restore clones that Patroni doesn’t manage (for example the postgres-main-v17-pitr-* nodes), and the archive and delayed clusters, are out of scope.
  • Rollback is a revert plus a chef run on the leader, or printf 'ignore_slots: null\n' | sudo gitlab-patronictl edit-config --apply - --force. Don’t roll back with a restart, switchover or pause/resume planned on that cluster: the Siphon slots are exposed again as soon as the matcher is gone.

After a switchover or failover needs a person. An unplanned failover, including a watchdog reset, has no one to run it. The candidate fix from production#23172 is to use Patroni’s on_role_change callback (today /bin/echo on_role_change-do_nothing) to run the guarded drop when a node becomes a replica, plus an alert on exiting from slot synchronization in the Postgres logs. This must cover every leader change, not only watchdog-driven ones. It is needed before the watchdog is enforced in gprd, and it is a Phase 2 gate of the gprd sec watchdog change (production#23457). Until then, the gprd watchdog change requests run the guarded drop by hand after each planned switchover.

Until gitlab-patroni!214 merges and a cluster’s role runs the recipe, the script is deployed by hand to the leader. That means no idempotent deployment, no failover safety and no log rotation; /tmp/siphon_slot_backup_refresh.log grows without limit. See Script for what the recipe changes.

Rollout order: gstg sec (chef-repo!8338), then gstg ci and main, then gprd, each after ignore_slots is live on that cluster. The recipe needs gitlab-patroni 1.17.0. On 2026-11-13, environments/gstg.json and environments/gprd.json in chef-repo pinned gitlab-patroni at = 1.13.0, so check how the new version reaches the nodes before merging a role change: a role that lists a recipe its pinned cookbook doesn’t have fails its chef run. A pin bump also brings in every cookbook change in between.

Planned follow-ups on the script: retry the create within the same run, and make CREATE_TIMEOUT settable per cluster. The script reports only through its log; there is no metric for “last successful refresh” or “backup slot missing” yet, so a cluster that keeps skipping or failing is only visible in the log and through the horizon alerts, once those exist.

The retired rotation script is still in the cookbook

Section titled “The retired rotation script is still in the cookbook”

files/default/manage_siphon_slots.sh is still on gitlab-patroni master. It implements approach 1: every hour it creates siphon_backup_slot_<HH> before dropping the slot from RETENTION_HOURS ago (default 6). gitlab-patroni!214 does not remove it. It should be deleted before someone finds it and wires it up.

runbooks!11389 added the dashboard, but none of these alerts exists yet. They are tracked in production-engineering#30301:

  • catalog_xmin age per slot, across all slots, including physical ones;
  • age(datfrozenxid) rising while pg_txid_xmin_age stays flat, which is the signature of this failure;
  • the age of inactive or orphaned slots.

The existing alerts don’t cover this:

  • PatroniXminAgeTooLarge (pg_txid_xmin_age) can’t see a slot pinning catalog_xmin.
  • The pg_xid_wraparound and pg_txid_vacuum_to_wraparound saturation alerts fire far too late.

A related enhancement is under discussion: an S1 alert on the primary Siphon slot, so that the on-call engineer is engaged the moment it fails rather than finding out from downstream staleness.

PG18 (sec) also has idle_replication_slot_timeout, which is off by default. The backup slot is always idle, so if this setting is ever enabled it must be well above the refresh interval. Set long, it would be a built-in backstop for orphaned slots. It also applies to the primary Siphon slot if Siphon is disconnected for that long. PG17 (main, ci) doesn’t have it.

False-positive lag alerts on the backup slot

Section titled “False-positive lag alerts on the backup slot”

SiphonLogicalReplicationSlotLagHigh (s2, pages Platform Insights) is defined on pg_replication_slots_confirmed_flush_lsn_bytes{slot_name=~".*siphon.*"} above 100 GiB for 5 minutes, on the primary only. It fires for each slot and can’t tell the slot in use from the backup slot. The backup slot is never consumed, so its lag grows until the next refresh, past 100 GiB on busy clusters. On gprd sec it fired for about 23 hours in total over 7 days, with a peak of 244 GiB.

The rule has no type label, so a silence can only match on slot_name and fqdn. An fqdn-pinned silence stops matching after a failover.

Fix in progress: runbooks!12101 (open) excludes slot_name=~".*siphon_backup.*" from the rule in gprd and gstg, so backup slots stop alerting and every other Siphon slot keeps the same threshold and routing. WAL retained by the backup slot is then covered only by the disk saturation alerts and the horizon alerts, once those exist.

The longer-term option is siphon#284 (open). Siphon 0.0.157-beta exports siphon_replication_slot_lag_bytes{app_id, slot_name} for the slot it is actually reading, and moving the alert to that metric would alert on whichever slot Siphon uses.

The PG17→PG18 upgrade of sec was the original motivation for this work. On gstg sec (2026-06-18, production#22298, root cause in dbo-issue-tracker#749), the steps were:

  1. The Siphon slot’s on-disk state (pg_replslot/<slot>/, with failover and synced set) had been captured in GCP disk snapshots.
  2. A node built from those snapshots came up still holding stg_sec_siphon_slot_1 with synced = t.
  3. PostgreSQL refuses to drop a synced slot while the node is in recovery.
  4. A cascading standby cannot run slot sync to clean it up.
  5. So the slot couldn’t be removed there, which blocked pg_upgrade.

The same workaround was then used for gprd sec in production#22366.

Backup slots reduce the blast radius, but they don’t make major upgrades hands-off. The complementary work is Siphon-side operational controls (pause/resume replication, clearing the LSN store), tracked in siphon#264.

Fallback: full rebuild when all slots are lost

Section titled “Fallback: full rebuild when all slots are lost”

If the primary slot and the backup slot are both gone or invalidated, follow the Siphon service runbook: recreate the slot and run consistency checks with corrections for the affected database. A full re-snapshot (1–2 weeks on production) is the last resort.

The Orbit cutover plan, confirmed with the Orbit team:

  1. Orbit keeps serving from its current ClickHouse instance throughout.
  2. The datalake is rebuilt in parallel through a full Siphon refresh.
  3. Orbit reindexes into the new instance and cuts over, which takes about 1.5–2 hours.

So the customer impact is stale data, not an outage.

Related work:

  • Consistency checks (shipped): siphon#276, “Source↔target completeness reconciliation (true-up & audit) for Siphon CDC”, closed 2026-10-19. The consistency checker compares PostgreSQL and ClickHouse range by range, and with corrections enabled re-reads the ranges that differ. This is what repairs the gap after a slot loss.
  • Parallel snapshot plus Siphon extraction on a large VM with Hyperdisk Extreme, to cut full-rebuild time. Needs a POC.
  • Narrowing the replicated data set so that a re-snapshot is smaller. Feasibility not yet assessed.
ReferenceWhat
gitlab-org/gitlab#605537Design discussion and decision record for all three approaches
production#22807 / #22817INC-13566 / INC-13574, the incidents that retired approach 1
production#22812 (confidential)Incident review
production#22859Advance-design production window (approach 2)
production#22939Single-slot advance tests, which retired approach 2
production#22298, #22366sec PG18 upgrades (synced-slot blocker)
production#23006First gprd sec refresh validation (2026-10-14)
production#23400gprd sec week-long refresh validation
production#23426gprd ci 24h refresh validation
gitlab-org/gitlab#605537 note_4198362282gprd sec and ci validation results, and the decision to use the gated refresh
production#23172, chef-repo!8244gstg ignore_slots, and the old-leader cleanup (step 3c)
production#23458, chef-repo!8309gprd ignore_slots on sec, ci and main
production#23352, #23457Patroni watchdog: gstg enforcement and the gprd sec rollout (switchover steps include the old-leader cleanup)
production-engineering#30300Tracking issue for this runbook
production-engineering#30301Alerting on slots pinning the catalog horizon
production-engineering#30302gstg backup slots missing or not refreshing
production-engineering#29661Orphaned backup slot lifecycle
gitlab-patroni!214Chef packaging of the refresh script
runbooks!11389Slot xmin age dashboard
runbooks!12101Exclude backup slots from SiphonLogicalReplicationSlotLagHigh
siphon#264Siphon API for pause/resume/clear LSN store
siphon#265Slot priority list (shipped)
siphon#276Source↔target completeness reconciliation (shipped)
siphon#284Adjust WAL alerts to account for backup slots
  • Backup slot, refresh job, Patroni, WAL, vacuum horizon: #g_database_operations (Database Operations).
  • Siphon behaviour, slot priority list, downstream data integrity: #g_analytics_platform_insights (Platform Insights).