Siphon Logical Replication Slot (LRS) Resilience
Summary
Section titled “Summary”Siphon consumes PostgreSQL WAL through a single logical replication slot per
database cluster (prd_main_siphon_slot_1, prd_ci_siphon_slot_1,
prd_sec_siphon_slot_1, and the stg_* equivalents). That slot is a single
point of failure. If it is dropped, invalidated, or lost during a major
PostgreSQL upgrade, Siphon can’t resume from where it was: every table of that
database has a data gap. Recovery means recreating the slot and running
consistency checks with corrections to repair the skipped rows, with a full
re-snapshot into ClickHouse as the worst case.
To remove that single point of failure, the Patroni leader keeps one backup
logical replication slot per cluster. A gated script drops and recreates it
every hour, so a recent, usable fallback slot always exists. Siphon is
configured with a replication.slot_priority_list. If the primary slot stops
working, Siphon switches to the backup slot rather than failing and forcing a
rebuild.
This runbook covers:
- the design and the automation that drops and recreates the backup slot
- how to verify that the automation is healthy, and which signal actually
matters (
catalog_xminage, not only WAL) - what to do when the primary Siphon slot is lost, and how to get back to the canonical slot name without a data gap
- the two retired designs and why each failed
- the fallback path when all slots are lost (full rebuild)
- the open gaps that still need closing
Owning teams: Database Operations owns the backup slot and its automation on Patroni. Platform Insights owns Siphon’s configuration and failover behaviour.
Origin: R&D Summit FY27, [Platform] Challenge #1: DIP everywhere.
Last reviewed: 2026-11-13.
Why this matters
Section titled “Why this matters”When a Siphon slot is lost, replication can’t resume from the slot’s position. No data is lost in PostgreSQL, but there is a gap in every table of that database: a few thousand rows if the slot was gone for seconds, gigabytes if it was gone for minutes. The Siphon service runbook covers the recovery: recreate the slot, then run consistency checks with corrections to repair the skipped rows. That takes a while. A full re-snapshot of the database’s tables (1–2 weeks on production) is the worst-case fallback, and everything depending on ClickHouse is stale until it finishes.
A usable backup slot avoids the gap entirely: Siphon re-delivers at most one refresh interval of WAL, which ClickHouse absorbs as duplicates.
A WAL archive is not a fallback. A new logical slot always starts at the current WAL position, so PITR gives Siphon no resume point.
Cluster capacity reference
Section titled “Cluster capacity reference”Measured from Mimir over the 14 days to 2026-11-07.
| Cluster | PG | Primary slot | WAL generation (avg / hourly p95 / max hour) | XID rate | max_slot_wal_keep_size |
|---|---|---|---|---|---|
sec | 18 | prd_sec_siphon_slot_1 | ~156 / ~221 / ~278 GB/h | ~0.9M/h | 1600 GiB |
main | 17 | prd_main_siphon_slot_1 | ~322 / ~433 / ~502 GB/h | ~10.9M/h | 3640 GiB |
ci | 17 | prd_ci_siphon_slot_1 | ~159 / ~222 / ~288 GB/h | ~18.9M/h | 6400 GiB |
The registry cluster has only physical slots and no Siphon.
The XID rate matters as much as the WAL rate. It decides how far an unconsumed
slot’s catalog_xmin falls behind between refreshes. ci uses XIDs about 20
times faster than sec.
Design: gated drop-and-recreate (“refresh”)
Section titled “Design: gated drop-and-recreate (“refresh”)”There is one backup slot per cluster. siphon_slot_backup_refresh.sh drops and
recreates it every hour. Each run:
- Takes a lock (gitlab-patroni!214 version only). An
flockonLOCK_FILEmakes an overlapping run exit cleanly. Hand-deployed copies don’t have it; see the note under Script. - Checks for the leader. The check is
SELECT NOT pg_is_in_recovery(). The script exits 0 on a standby. In the gitlab-patroni!214 version, a failed psql call exits 1 instead. - Requires the primary Siphon slot to exist (gitlab-patroni!214 version).
If it doesn’t, Siphon isn’t set up on this cluster or
ENV/TYPEare wrong, so the run aborts with exit 1 before touching any slot. - Recreates a missing backup slot straight away (gitlab-patroni!214 version). If the backup slot doesn’t exist, for example because an earlier run died between drop and create, or because the node was just promoted and its copy wasn’t sync-ready, the script creates it now, without the WAL gate, and stops. Creating a slot never touches the primary slot.
- Runs the health gate on the primary Siphon slot. The slot must be
active, havewal_status = reserved, and be neither conflicting nor invalidated. The script then samples the slot’s retained WAL (pg_current_wal_lsn() − restart_lsn) everySAMPLE_EVERYseconds (10) forTHRESHOLD_INTERVALseconds (30). If the mean is aboveTHRESHOLD_MB, it skips the run. A missing primary slot aborts with exit 1. Any other gate failure logsABORT/SKIPand exits 0. In every case the existing backup slot is left untouched. - Drops the backup slot. It refuses if the backup slot is
active, meaning a consumer is attached, as happens after Siphon has failed over to it. The run then aborts without recreating. - Waits
DROP_RECREATE_DELAYseconds. Every current cron uses 60; the script default is 0. See Why the delay. - Creates the new backup slot with
pg_create_logical_replication_slot('<backup>', 'pgoutput', false, false, true), that is,failover = true. In the gitlab-patroni!214 version the create is bounded bystatement_timeout = CREATE_TIMEOUT(300 s), and the script then checks that the slot exists withfailover = t. A failure after the drop exits non-zero and logsbackup slot ... is MISSING; the next run recreates it (step 4). - Logs every step. This includes before and after snapshots of every slot’s
catalog_xminandage(datfrozenxid), plus acatalog_xminline every 15 s during the wait (every 30 s in hand-deployed copies).
Why this works where the earlier designs did not
Section titled “Why this works where the earlier designs did not”- No decoding at all. The backup slot is never read, so the design does not depend on decode throughput. The advance-based design did, and it fell behind on high-WAL-rate clusters.
- The horizon releases. Dropping the only unconsumed backup slot
recomputes the cluster-wide slot horizon straight away
(
ReplicationSlotsComputeRequiredXmin, which the drop calls). Vacuum and the next slot creation both see the horizon fall back to the healthy primary slot’s position. - It is instant. No WAL is scanned, so it works at any WAL rate.
- The primary slot is only read. It is never dropped, advanced or changed.
Why the delay
Section titled “Why the delay”On the primary, the drop releases the horizon immediately. The delay is for the standbys:
- The backup slot has
failover = true, so each standby that syncs slots holds a copy of it. - Through
hot_standby_feedback, the standby keeps reporting that copy’s oldcatalog_xminback to the primary on its physical slot. - It keeps doing so until its slot-sync worker notices the drop (up to 30 s)
and sends fresh feedback (
wal_receiver_status_interval, 10 s by default). - A slot created within that window would inherit the old horizon through the standby’s physical slot.
The fixed 60 s covers this window. Polling until the physical slots’
catalog_xmin has caught up would be more robust; see
Open gaps.
What it costs
Section titled “What it costs”- Retained WAL: about refresh interval × WAL rate. At hourly cadence that is
roughly 150–280 GB on sec and ci, and up to ~500 GB on main. It is bounded by
max_slot_wal_keep_size. - Catalog cleanup lag: about one interval of XIDs. Measured peaks just
before a refresh are ~1.6M on sec and ~15–19M on ci. That is under 1% and
about 9% of the 200M
autovacuum_freeze_max_agetrigger. A skipped refresh roughly doubles the peak, and consecutive skips keep adding to it. - Time with no backup slot: each run spends
DROP_RECREATE_DELAYplus slot creation time without one. Creation waits for every transaction holding an XID at that moment to finish. Read-only queries don’t block it, but a long-running write transaction can block it for minutes. - Fallback window: if the primary slot is lost just before a refresh, Siphon has at most one interval of WAL to replay.
Slot naming
Section titled “Slot naming”<env>_<db>_siphon_backup_slot_01| Cluster | Primary slot | Backup slot |
|---|---|---|
gprd sec | prd_sec_siphon_slot_1 | prd_sec_siphon_backup_slot_01 |
gprd ci | prd_ci_siphon_slot_1 | prd_ci_siphon_backup_slot_01 |
gprd main | prd_main_siphon_slot_1 | prd_main_siphon_backup_slot_01 |
gstg sec / main / ci | stg_<db>_siphon_slot_1 | stg_<db>_siphon_backup_slot_01 |
The script only ever creates ..._backup_slot_01. If a cluster has more than
one *_siphon_backup_slot_* slot, something else is creating them, and the
cluster is in the INC-13574 failure mode. See
Emergency: backup slots pinning the catalog horizon.
The automation
Section titled “The automation”Script
Section titled “Script”siphon_slot_backup_refresh.sh has so far been deployed by hand to
/var/opt/gitlab/postgresql/siphon_slot_backup_refresh.sh, owned by
gitlab-psql, mode 0750.
gitlab-patroni!214
(in review) packages it as the gitlab-patroni::siphon_slot_backup_refresh
recipe, and
chef-repo!8338
adds that recipe to gstg-base-db-patroni-sec-v18 first. Once a role runs the
recipe:
- the script is at
/usr/local/bin/siphon_slot_backup_refresh.shand logs to/var/log/gitlab/postgresql/siphon_slot_backup_refresh.log, rotated daily (7 kept); - the cron is on every member of the cluster, and the leader check makes it a no-op on standbys, so it follows the leader after a failover;
- settings come from role attributes under
gitlab-patroni.siphon_slot_backup_refresh(env,type,threshold_interval,threshold_mb,sample_every,drop_recreate_delay,plugin,cron.minute/hour). The recipe passes them on the cron command line, not as crontab variables, so they don’t leak into other cron entries; - with
enabled: false, orenv/typeunset, the recipe removes the cron, script and logrotate entry. Chef doesn’t undo a recipe that is taken out of the run list, so to take the automation off a cluster, setenabled: falseand converge first, then remove the recipe; - Chef owns the crontab entry. A hand edit (for example commenting it out to pause it) is put back on the next chef run, within about 30 minutes. See Disable the automation on a cluster.
| Variable | Required | Meaning | Values in use |
|---|---|---|---|
ENV | yes | First part of the slot name | prd / stg |
TYPE | yes | Second part of the slot name | main / ci / sec |
THRESHOLD_MB | yes | Mean retained WAL on the primary slot above which the run is skipped | 1500 (gstg sec, chef-repo!8338), 3000 (gprd sec test), 3500 (gprd ci test); Chef default 2000 |
THRESHOLD_INTERVAL | yes | Seconds over which retained WAL is sampled | 30 |
SAMPLE_EVERY | no | Seconds between samples | 10 (default) |
DROP_RECREATE_DELAY | no | Seconds between drop and recreate | 60 (Chef default and every cron so far; script default 0) |
CREATE_TIMEOUT | no | statement_timeout for the slot create, in seconds (gitlab-patroni!214 only) | 300 (default; the recipe doesn’t pass it yet) |
PLUGIN | no | Output plugin | pgoutput (default) |
LOG_FILE | no | Log path | Chef: /var/log/gitlab/postgresql/siphon_slot_backup_refresh.log; script default /tmp/siphon_slot_backup_refresh.log |
LOCK_FILE | no | flock path (gitlab-patroni!214 only) | The script file itself (default), opened read-only, so any user that can read the script can take the lock |
Hourly at minute 0. With Chef, the entry is on every member and looks like this (gstg sec, chef-repo!8338):
# Chef Name: siphon_slot_backup_refresh0 * * * * ENV=stg TYPE=sec THRESHOLD_INTERVAL=30 THRESHOLD_MB=1500 SAMPLE_EVERY=10 DROP_RECREATE_DELAY=60 PLUGIN=pgoutput LOG_FILE=/var/log/gitlab/postgresql/siphon_slot_backup_refresh.log /usr/local/bin/siphon_slot_backup_refresh.sh > /dev/null 2>> /var/log/gitlab/postgresql/siphon_slot_backup_refresh.logThe hand-deployed entries below, as gitlab-psql on the leader only, were used
in the gprd validation tests. Both tests have ended (see
Current status):
# Siphon slot backup refresh - gprd sec (production#23400)0 * * * * ENV=prd TYPE=sec THRESHOLD_INTERVAL=30 THRESHOLD_MB=3000 DROP_RECREATE_DELAY=60 /var/opt/gitlab/postgresql/siphon_slot_backup_refresh.sh >> /tmp/siphon_slot_backup_refresh.log 2>&1
# Siphon slot backup refresh - gprd ci (production#23426)0 * * * * ENV=prd TYPE=ci THRESHOLD_INTERVAL=30 THRESHOLD_MB=3500 DROP_RECREATE_DELAY=60 /var/opt/gitlab/postgresql/siphon_slot_backup_refresh.sh >> /tmp/siphon_slot_backup_refresh.log 2>&1The script already tees to LOG_FILE, so with this redirect each line
probably shows up twice in the log. The Chef cron in gitlab-patroni!214 sends
stdout to /dev/null instead.
Interval: hourly. Platform Insights confirmed that 1h and 2h both work for
Siphon, and 1h was chosen. A shorter interval lowers the catalog_xmin peak
between refreshes and gives a shorter fallback window. A longer one does the
reverse.
Siphon side: slot priority list
Section titled “Siphon side: slot priority list”This is set in the ArgoCD applications repo
(gitlab-com/gl-infra/argocd/apps),
services/siphon/env/orbit-<env>/values.yaml, under each producer deployment:
deployments: postgres-sec-producer: appConfig: replication: publication_name: stg_sec_siphon_publication_1 slot_name: stg_sec_siphon_slot_1 slot_priority_list: - stg_sec_siphon_backup_slot_01ArgoCD auto-sync is off for Siphon, so a values change needs a manual sync.
How Siphon chooses a slot (SetupReplicationSlot):
- It walks
[slot_name] + slot_priority_listin order. A slot is skipped if it is missing, haswal_status = 'lost', hasconflicting = true, or hasinvalidation_reasonset. Siphon connects to the first slot that works and resumes from that slot’sconfirmed_flush_lsn. - Selection runs when the producer starts. A dropped or invalidated slot makes replication fail, and the producer then restarts in-process (no pod restart) and selects again. That is how failover happens.
- A plain reconnect keeps the slot already chosen. Once Siphon is on the backup slot it stays there, even after the primary slot is recreated. Switching back needs a deliberate restart (see Recovery).
- If no slot works, the producer logs
no functional replication slot found among [...]and keeps restarting.
Current status (2026-11-13)
Section titled “Current status (2026-11-13)”| Environment | Refresh script + cron | Siphon slot_priority_list | Notes |
|---|---|---|---|
gstg sec | ⚠️ Not refreshing since the 2026-11-04 failover test (production-engineering#30302). The backup slot’s catalog_xmin age was ~1.4M and still climbing on 2026-11-12 | ✅ | Failover validated end to end (2026-11-04). First Chef rollout target: chef-repo!8338 (THRESHOLD_MB=1500) |
gstg main | ⚠️ Backup slot missing since a leader change on 2026-10-27 | ✅ | This is the failover-orphan gap |
gstg ci | ⚠️ Backup slot missing since a leader change on 2026-10-27 | ✅ | Same |
gprd sec | ✅ Validated over a week (2026-11-03 to 11-10), then removed. The backup slot’s metrics end at ~2026-11-10 19:00 UTC | ❌ | production#23400, THRESHOLD_MB=3000, leader patroni-sec-v18-101. No backup slot until the Chef rollout |
gprd ci | ✅ Validated, then removed (2026-11-06 to 11-08) | ❌ | production#23426, THRESHOLD_MB=3500, leader patroni-ci-v17-27. No backup slot until the Chef rollout |
gprd main | ❌ Not scheduled | ❌ | Highest WAL rate; needs its own threshold measurement |
| Chef packaging | 🟡 In review: gitlab-patroni!214 (recipe, cookbook 1.17.0) and chef-repo!8338 (gstg sec role) | n/a | Check the gitlab-patroni pin before the role change (see Not yet packaged in Chef). Then gstg ci and main, which also fixes the gstg rows above, then gprd |
Patroni ignore_slots | ✅ gstg sec, ci, main v18 · ✅ gprd sec-v18, ci-v17, main-v17 | n/a | gprd done 2026-11-16 in production#23458 (~C2, chef-repo!8309). Run the old-leader cleanup after every leader change. See Patroni and the Siphon slots |
| Backup-slot lag alert exclusion | 🟡 runbooks!12101 (open) | n/a | See False-positive lag alerts on the backup slot |
Validation record
Section titled “Validation record”gprd sec refresh, 2026-10-14 21:00 to 10-15 ~15:00 UTC
(production#23006,
THRESHOLD_MB=1500):
- 18 recreates in 18 hours, with no skipped runs.
- The backup slot’s
catalog_xminage dropped to ~35–80K right after each recreate and climbed to ~0.6–1.6M by the next run. The peak over the whole window was ~1.6M, under 1% of the 200M anti-wraparound trigger. - The primary slot
prd_sec_siphon_slot_1was unaffected throughout (~2.5K–25K). - Teardown was clean. After the test the backup slot was dropped and the horizon went back to baseline (~13K).
datfrozenxidstayed at ~10% of the wraparound budget throughout.
Separate testing showed the gate skipping refreshes during the recurring daily primary-slot spike and resuming afterwards.
gprd ci refresh, 2026-11-06 21:05 to 11-08 ~19:30 UTC
(production#23426,
THRESHOLD_MB=3500):
- 46 refreshes from 11-06 22:02 to 11-08 19:02 UTC, one per hour, with no skipped runs. The window was planned as 24 hours and ran about 46.
- The backup slot’s
catalog_xminage dropped to ~0.1–0.7M after each refresh and reached ~7–19M before the next. That is about 10 times sec’s sawtooth, because ci burns ~19M XIDs/h against ~1M/h on sec, and still under 10% of the 200M anti-wraparound trigger. - The primary slot
prd_ci_siphon_slot_1stayed healthy throughout (catalog_xminage ≤ 0.31M). pg_xid_wraparoundsaturation on ci stayed at 9–13%, the same as the week before.- Peak backup-slot lag was 153 GiB, 2.4% of ci’s
max_slot_wal_keep_size(6.25 TiB). - Teardown was clean: the backup slot was dropped and the cron removed.
SiphonLogicalReplicationSlotLagHighfired on the backup slot again on 11-08, after the CR’s 26h silence had expired. See Open gaps.
gprd sec refresh, one week, 2026-11-03 22:05 to 11-10 ~19:00 UTC
(production#23400,
THRESHOLD_MB=3000; results and graphs in
gitlab-org/gitlab#605537):
- The hourly sawtooth held for the whole week: the backup slot’s
catalog_xminage dropped to ~60–250K after each refresh and was normally ~0.3–1.6M before the next one. - The gate skipped refreshes only while the primary slot was behind, during the daily spikes on 11-06 (~15:30–17:00 UTC) and 11-09 (~00:30–02:00 UTC), and resumed on its own. The highest point of the week, ~2.9M on 11-06, came from those skips. That is under 2% of the 200M anti-wraparound trigger.
- The primary slot
prd_sec_siphon_slot_1stayed at ~1–25K outside those spikes (spike peaks ~0.5M and ~0.8M). pg_xid_wraparoundsaturation on sec stayed at 9.1–9.4% all week.- The backup slot was dropped at the end of the test: its series in Mimir end at
about 11-10 19:00 UTC, and only
prd_sec_siphon_slot_1remains on the four members. - With these results, the gated drop-and-recreate refresh is the agreed design (decision).
Siphon failover, gstg sec, 2026-11-04 (image
siphon:0.0.156-beta, postgres-sec-producer;
record):
| Time (UTC) | Event |
|---|---|
| 10:21:29 | Terminated the walsender and dropped stg_sec_siphon_slot_1. Last acked LSN B80B/EA000060 |
| 10:21:31 | replication slot not functional, trying next, then using replication slot stg_sec_siphon_backup_slot_01, resuming from B80B/D426AD50 |
| 10:21:41 | Past the old stop point; caught up in under 10 s |
| 10:23:51 | Recreated stg_sec_siphon_slot_1 |
| 10:24:01 | Backup slot past the new slot’s start LSN, so safe to restart |
| 10:25:11 | After a pod restart, using replication slot stg_sec_siphon_slot_1 |
Failover took ~1.6 s. The backup slot was ~350 MiB of WAL behind, and Siphon
replayed that in under 10 s. Most of that WAL was likely for tables outside the
publication, which pgoutput skips. Switching back replayed ~16 MiB of
deliberate overlap. In both directions the resume LSN was behind the last
acked LSN, so nothing was skipped. ClickHouse’s ReplacingMergeTree absorbs the
duplicates.
An earlier failover test on 2026-08-27 showed the same behaviour under the retired multi-slot configuration.
Verification
Section titled “Verification”The signal that matters: catalog_xmin age
Section titled “The signal that matters: catalog_xmin age”WAL retention is the obvious cost of a slot, but it is not the dangerous
one, because max_slot_wal_keep_size bounds it. Nothing bounds the horizon
directly, and a pinned horizon is what caused INC-13574. Check it first, across
all slots, physical included:
SELECT slot_name, slot_type, active, synced, temporary, age(catalog_xmin) AS catalog_xmin_age, age(xmin) AS xmin_age, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)) AS retained_wal, wal_statusFROM pg_replication_slotsORDER BY greatest(age(catalog_xmin), age(xmin)) DESC NULLS LAST;A healthy leader, partway through a run, shows:
- exactly one
*_siphon_backup_slot_01row, withactive = f; - the backup slot’s
catalog_xmin_agemoving in a sawtooth that never climbs steadily, within the ranges below; - the primary slot in the range below;
- physical slots (standbys) low as well, since they carry standby feedback;
wal_status=reservedorextendedon both Siphon slots.unreservedmeans the slot loses WAL at the next checkpoint, so act now.
| Cluster | Backup slot, just after a refresh | Backup slot, just before the next | Primary slot |
|---|---|---|---|
| sec | ~10–80K | ~0.8–1.6M | low tens of thousands |
| ci | ~0.2–0.4M | ~15–19M | ~0.1–0.3M |
| main | TBD | TBD | normally ~0.1–0.3M, with periodic spikes |
Escalate if catalog_xmin_age on any slot keeps climbing without dropping back,
or if age(datfrozenxid) is rising while pg_txid_xmin_age stays flat. That
divergence is the INC-13574 signature.
Metrics: pg_replication_slots_catalog_xmin_age,
pg_replication_slots_xmin_age, pg_txid_xmin_age,
pg_database_wraparound_age_datfrozenxid. wal_status is not exported, so
check it with SQL.
On standbys: is the backup slot sync-ready?
Section titled “On standbys: is the backup slot sync-ready?”failover = true only marks the slot for synchronization (PG17+). It reaches a
standby only if that standby has sync_replication_slots = on,
hot_standby_feedback = on, primary_slot_name set, and dbname in
primary_conninfo, and streams directly from the primary. Cascading standbys
cannot sync. After a failover, the slot is usable only if, before the failover,
it showed synced = t and temporary = f on the standby that was promoted.
A slot that is never consumed may stay temporary = t on a standby, so check:
-- on each replicaSELECT slot_name, synced, temporary, invalidation_reasonFROM pg_replication_slots WHERE slot_name LIKE '%siphon%';Our clusters use PostgreSQL’s native slot sync (sync_replication_slots = on
in the Patroni roles), not Patroni’s permanent slots. Patroni is told to leave
the Siphon slots alone with ignore_slots (see
Patroni and the Siphon slots).
Dashboard
Section titled “Dashboard”PostgreSQL replication slot xmin age
was built for this failure mode after INC-13566
(runbooks!11389).
It puts the whole chain of signals on one page: slot catalog_xmin age, cluster
xmin age, datfrozenxid age, autovacuum worker saturation and analyze rate.
Also useful:
Is the job running?
Section titled “Is the job running?”sudo gitlab-patronictl list # confirm this node is the leadersudo grep -r siphon_slot_backup_refresh /var/spool/cron/crontabs/ # whichever user owns itsudo tail -50 /var/log/gitlab/postgresql/siphon_slot_backup_refresh.log # Chefsudo tail -50 /tmp/siphon_slot_backup_refresh.log # hand-deployedA healthy log on the leader shows === Done === once an hour, or an explicit
reason when a run does nothing:
SKIP: mean WAL ... > threshold: normal during the daily spike;ABORT: primary slot ... does not exist/is NOT active/wal_status=.../conflicting/invalidated: the primary slot is unhealthy. Check Siphon;Not the leader node: normal on standbys, where the Chef cron also runs;Another refresh run holds ...: runs overlapped (gitlab-patroni!214 version only);Backup slot ... is MISSING - recreating it now: the previous run failed after the drop, or the node was just promoted without a sync-ready copy. One occurrence is fine;FAILED (exit N): backup slot ... is MISSING: the create failed or timed out (CREATE_TIMEOUT, usually a long-running write transaction). The cluster has no backup slot until the next run. If it repeats, check for long transactions and create the slot by hand.
Consecutive skips or aborts over many hours are not normal. They mean the backup slot is ageing with nothing resetting it.
Recovery: the primary Siphon slot is lost
Section titled “Recovery: the primary Siphon slot is lost”The examples below use gprd sec. Substitute <env>_<db> for other clusters.
Symptoms: the Siphon producer restarts, and the primary slot is either
missing from pg_replication_slots or present with wal_status = 'lost',
conflicting = true, or invalidation_reason set.
What happens automatically (only where slot_priority_list is configured,
currently staging only): the producer restarts in-process, walks the slot list
and connects to the backup slot within seconds. It replays at most one refresh
interval of WAL. No re-snapshot is required.
Step 1: Confirm which slot Siphon is using
Section titled “Step 1: Confirm which slot Siphon is using”SELECT slot_name, active, active_pid, wal_status, age(catalog_xmin) AS catalog_xmin_age, pg_size_pretty(pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn)) AS lagFROM pg_replication_slotsWHERE slot_name ILIKE '%siphon%'ORDER BY active DESC, slot_name;The row with active = t and a live active_pid is the one in use. Confirm it
in the producer logs:
glsh kube use-cluster orbit-prd --no-proxy # orbit-stg for stagingkubectl logs -n siphon -l app=postgres-<db>-producer --tail=200 | grep -E "replication slot"Step 2: Pause the refresh job
Section titled “Step 2: Pause the refresh job”The script should not touch the backup slot while Siphon is using it. It aborts when the primary slot is missing or inactive, and it refuses to drop an active backup slot. Pause it anyway, so that no refresh runs while you recreate and switch slots.
-
Hand-deployed: as
gitlab-psqlon the leader,crontab -eand comment out thesiphon_slot_backup_refreshline. -
Chef-managed: a crontab edit is put back by the next chef run. On every member (the cron runs wherever the leader is), disable chef with a reason first, then comment out the line:
Terminal window sudo chef-client-disable "Siphon slot recovery, <issue link>"sudo crontab -e -u <cron user> # comment out the siphon_slot_backup_refresh lineRe-enable chef in Step 7. For a longer pause, set
enabled: falsein the role instead (see Disable the automation on a cluster).
Step 3: Stabilise
Section titled “Step 3: Stabilise”Confirm that lag on the slot now in use is falling. If it isn’t, treat this as a high replication lag incident first and come back to the restore afterwards.
Step 4: Recreate the canonical slot
Section titled “Step 4: Recreate the canonical slot”SET statement_timeout = '10min'; -- creation waits for in-progress write transactionsSELECT * FROM pg_create_logical_replication_slot( 'prd_sec_siphon_slot_1', -- slot name 'pgoutput', -- plugin false, -- temporary false, -- twophase true -- failover);A cancelled create leaves no slot behind, so retrying is safe. Keep
failover = true: it marks the slot for sync to the standbys (see
sync-ready).
Step 5: Wait for overlap before switching back
Section titled “Step 5: Wait for overlap before switching back”Wait until the backup slot that Siphon is consuming has reached the new slot’s
confirmed_flush_lsn:
SELECT s1.slot_name AS target_slot, s1.confirmed_flush_lsn AS target_lsn, s2.slot_name AS backup_slot, s2.confirmed_flush_lsn AS backup_lsn, (s1.confirmed_flush_lsn <= s2.confirmed_flush_lsn) AS safe_to_switch, pg_wal_lsn_diff(s2.confirmed_flush_lsn, s1.confirmed_flush_lsn) AS overlap_bytesFROM pg_replication_slots s1CROSS JOIN pg_replication_slots s2WHERE s1.slot_name = 'prd_sec_siphon_slot_1' AND s2.slot_name = 'prd_sec_siphon_backup_slot_01';Proceed only when safe_to_switch = t, which usually takes a few minutes.
overlap_bytes is the amount Siphon will replay. That is harmless, because
ReplacingMergeTree deduplicates it downstream. This check assumes Siphon
acknowledges an LSN only after it has durably published everything before it.
Step 6: Restart Siphon onto the canonical slot
Section titled “Step 6: Restart Siphon onto the canonical slot”Siphon stays on the backup slot until it is restarted:
kubectl rollout restart deployment/postgres-<db>-producer -n siphonkubectl logs -n siphon -l app=postgres-<db>-producer --tail=100 \ | grep -E "using replication slot|resuming from replication slot LSN"Confirm that it picked the canonical slot with no not functional warning.
Step 7: Re-enable the refresh job
Section titled “Step 7: Re-enable the refresh job”Uncomment the cron entry, or on Chef-managed nodes run sudo chef-client-enable
and sudo chef-client, which restores it. Then run the script once by hand
(see Manual operations). From the log and from
pg_replication_slots, confirm that the backup slot was recreated and that its
catalog_xmin_age is back near the baseline for the cluster.
Manual operations
Section titled “Manual operations”Run a refresh by hand
Section titled “Run a refresh by hand”On a Chef-managed node, copy the exact command from the crontab entry
(sudo grep -r siphon_slot_backup_refresh /var/spool/cron/crontabs/) and run it
as the user that owns that crontab, so the settings match the role.
Hand-deployed:
sudo su - gitlab-psqlENV=prd TYPE=sec THRESHOLD_INTERVAL=30 THRESHOLD_MB=3000 DROP_RECREATE_DELAY=60 \ /var/opt/gitlab/postgresql/siphon_slot_backup_refresh.shDon’t run it within a few minutes of the top of the hour. Hand-deployed
copies have no overlap guard, so a manual run and the cron run could both drop
and recreate the slot at once. The gitlab-patroni!214 version takes an flock
on the script file itself, opened read-only, so the second run exits cleanly
whichever user started the first.
Create the backup slot by hand
Section titled “Create the backup slot by hand”SELECT * FROM pg_create_logical_replication_slot( 'prd_sec_siphon_backup_slot_01', 'pgoutput', false, false, true);Drop the backup slot
Section titled “Drop the backup slot”This is safe only when the slot is not in use. If Siphon has failed over, the backup slot is the one being consumed, and dropping it forces a full re-snapshot. Check first:
SELECT slot_name, active, active_pidFROM pg_replication_slots WHERE slot_name = 'prd_sec_siphon_backup_slot_01';-- only if active = f:SELECT pg_drop_replication_slot('prd_sec_siphon_backup_slot_01');Drop it on the primary. On a standby, synced slots can’t be dropped by hand; the slot-sync workers remove their copies.
After a switchover or failover: clean up the old leader
Section titled “After a switchover or failover: clean up the old leader”Do this after every leader change on a cluster with Siphon slots: planned switchovers, manual failovers, and unplanned failovers (including a watchdog reset).
Why. The old leader created the Siphon slots while it was the primary, so
its copies have synced = f. After it is demoted:
- PostgreSQL’s slot-sync worker stops at a non-synced slot with the same name:
exiting from slot synchronization because same name slot "<name>" already exists on the standby. This is the same on PG17 and PG18 (PG17 slotsync.c, PG18 slotsync.c). The node has no sync-ready copy, so a later promotion of it brings back a stale Siphon slot. - The stale slots never advance. They retain WAL on the old leader, and with
hot_standby_feedback = onthe standby reports theircatalog_xminto the new primary, which can hold back the new primary’s catalog horizon. - Patroni no longer drops them, because of
ignore_slots.
A node that was itself promoted from a standby can already have synced = t
copies (gstg main-103 on 2026-11-04). Then there is nothing to clean up.
The same applies to a replica built from a snapshot of the leader’s disk.
The snapshot carries the leader’s own synced = f Siphon slots, so the new
replica’s slot sync stops at them, exactly as on a demoted leader. Before
ignore_slots, Patroni dropped them when it started on the new node; with
ignore_slots, nothing does. Prefer snapshots taken from a replica, whose
copies have synced = t. If the source was the leader, run the guarded drop
below on the new replica once it is streaming.
Steps:
-
On the new leader first: check that the primary Siphon slot is there.
Terminal window sudo gitlab-psql -AtXc "select slot_name, active, failover, confirmed_flush_lsn from pg_replication_slots where slot_name like '%_siphon_slot_1'"If
*_siphon_slot_1is missing, the promoted node had no sync-ready copy. Stop here and don’t drop anything. Go to Recovery: the primary Siphon slot is lost. The old leader’s stale copy is then the only record of Siphon’s last position before the leader change, and the LSN-overlap check needs it. Keep it until Recovery is done. -
On the old leader, once it is streaming as a replica: confirm the error, and list what the drop would remove with its LSNs. Paste the output into the CR or incident, so the positions are kept after the drop:
Terminal window sudo grep "exiting from slot synchronization because same name slot" /var/log/gitlab/postgresql/*.csv | tail -2sudo gitlab-psql -AtXc "select slot_name, synced, active, restart_lsn, confirmed_flush_lsn from pg_replication_slots where slot_type = 'logical' and plugin = 'pgoutput' and database = 'gitlabhq_production' and not synced and not active and pg_is_in_recovery()" -
Drop them. The guard only acts on a standby (
pg_is_in_recovery()), and only on non-synced, inactivepgoutputslots, so on the primary it returns no rows:Terminal window sudo gitlab-psql -AtXc "select slot_name, pg_drop_replication_slot(slot_name) from pg_replication_slots where slot_type = 'logical' and plugin = 'pgoutput' and database = 'gitlabhq_production' and not synced and not active and pg_is_in_recovery()" -
Success: within about a minute the slot-sync worker recreates the Siphon slots on that node with
synced = tandtemporary = f, and no newexiting from slot synchronizationlines appear. On the new primary, the old leader’s physical slot shows acatalog_xminage in the same range as the other replicas. A never-consumed backup slot can staytemporary = tfor a while (see On standbys). -
On the new leader: check that the Siphon consumer has reconnected (
active = ton*_siphon_slot_1), then follow Patroni failover orphans the backup slot for the backup slot and the refresh cron.
This is the same guarded drop used in production#23172 (step 3c) and production#23352 (step 7), where it removed exactly the stale Siphon slots on gstg sec and ci.
Disable the automation on a cluster
Section titled “Disable the automation on a cluster”- Stop the cron:
- Chef-managed: set
"enabled": falseundergitlab-patroni.siphon_slot_backup_refreshin the cluster’s role, merge, and runsudo chef-clienton every member. The recipe removes the cron, the script and the logrotate entry. Only then remove the recipe from the run list, if wanted; removing it first leaves the cron behind. - Hand-deployed: comment out or remove the cron entry, as
gitlab-psql.
- Chef-managed: set
- Drop the backup slot on the leader. Check
active = ffirst, as above. Disabling the recipe does not drop the slot. - Confirm that
catalog_xminage is back to baseline. On the primary it happens within seconds. Standbys’ physical slots may report the old value for another ~30–40 s. - Expire any Alertmanager silence that was created for the slot.
Emergency: backup slots pinning the catalog horizon
Section titled “Emergency: backup slots pinning the catalog horizon”Signature:
age(datfrozenxid)is rising whilepg_txid_xmin_agestays flat;- one or more
*_siphon_backup_slot_*slots have a highcatalog_xmin_agethat only climbs; - autovacuum workers are saturated on wraparound-prevention vacuums;
- dead tuples are piling up, and the analyze rate is zero.
Act immediately: this escalates to user-visible impact.
-
Stop whatever is creating slots: comment out the cron entry.
-
Drop every unconsumed backup slot. The horizon only releases once every slot older than the healthy primary slot is gone.
SELECT slot_name, active, age(catalog_xmin)FROM pg_replication_slotsWHERE slot_name LIKE '%siphon_backup_slot%';SELECT pg_drop_replication_slot('<each slot with active = f>'); -
Confirm that
catalog_xminage drops to baseline; this is immediate on the primary.age(datfrozenxid)follows once autovacuum has finished its anti-wraparound passes over the catalogs. -
Autovacuum resumes on its own, but statistics are slow to catch up. Run a manual
ANALYZEon the busy tables that were starved. In INC-13566 it was the stale statistics, not the dead tuples themselves, that flipped query plans and caused the 500s. -
Escalate to
#g_database_operations.
Retired approaches (read before changing anything)
Section titled “Retired approaches (read before changing anything)”Approach 1: three hourly-rotated backup slots ❌
Section titled “Approach 1: three hourly-rotated backup slots ❌”What it was (Aug 2026): several backup slots per cluster. A new one was created every hour, and the stale one dropped a few hours later, so Siphon always had a slot at most a few hours old. It ran on staging from 2026-08-10 and was enabled on gprd sec.
What happened: INC-13566 (Sev2, production#22807) and INC-13574 (Sev4, production#22817), gprd sec, 2026-08-25 to 08-27, with 3h 52m of user impact. The combined review is production#22812 (confidential).
- From about 2026-08-25 13:45 UTC,
age(datfrozenxid)on patroni-sec-v18 rose from its ~200M baseline to ~266M by 08-27, in lockstep across all three databases.pg_txid_xmin_agestayed at 505 XIDs, while the four backup slots showed acatalog_xminage of about 265.9M. - Autovacuum stopped making progress across the instance. Dead tuples piled up on the largest, busiest tables, and some had not been vacuumed or analyzed for days.
- Stale statistics flipped the pipeline security tab queries to parallel sequential scans. CPU saturated, and there were 500s from 08:20 to 11:51 UTC on 08-27.
- Each backup slot also held 150–230 GB of WAL during its 3-hour life, about 600 GB pinned at once.
- Recovery: a manual
ANALYZEon the busy tables that had been starved, and dropping all four backup slots. The horizon released immediately,age(datfrozenxid)returned to baseline once autovacuum caught up, and autovacuum resumed. - gprd main showed the same climbing pattern, but it had enough headroom that it never became user-visible.
Why rotation cannot work (checked against the PostgreSQL source):
- A logical slot’s
catalog_xminmoves only when WAL is decoded through the slot and a confirmed position is recorded (LogicalConfirmReceivedLocation). A slot nobody reads never advances, however young it is. - A new logical slot is born already old.
pg_create_logical_replication_slot()sets the startingcatalog_xmintoGetOldestSafeDecodingTransactionId(catalogOnly = true). That is the oldest of the next XID, any running transaction’s XID, and thexmin/catalog_xminof every existing slot, physical slots included. So every hourly slot inherited the frozen horizon of the oldest unconsumed slot still alive. Rotation moved the floor forward only by the gap between consecutive creations, while XIDs advanced at production rate. - What
catalog_xminactually pins. A logical slot’scatalog_xminholds back cleanup of system catalogs (including shared catalogs) anduser_catalog_tabletables only, not ordinary tables. This has been true since logical decoding arrived in 9.4. Since PG14 it is implemented inComputeXidHorizons. The chain from there to user impact:- The slot horizon is a single cluster-wide value, applied to the catalogs of
every database whichever database the slot belongs to. So no catalog
anywhere could have its
relfrozenxidadvanced past it. datfrozenxidis capped by the same horizon (vac_update_datfrozenxid), so all three databases aged in lockstep.- Once they passed
autovacuum_freeze_max_age(200M), every catalog qualified for an anti-wraparound vacuum on every pass, and none could finish it. - While any database is past that limit, the autovacuum launcher sends each
new worker to the database with the oldest
datfrozenxidand ignores the rest (do_start_worker). - Ordinary tables were never blocked. Their dead tuples grew because no worker got to them, so stale statistics followed, then plan flips, then CPU saturation.
- The slot horizon is a single cluster-wide value, applied to the catalogs of
every database whichever database the slot belongs to. So no catalog
anywhere could have its
- Two symptoms, one cause. The same unconsumed slots pin WAL retention
(
restart_lsn) and the horizon (catalog_xmin).max_slot_wal_keep_sizelooks only atrestart_lsn. A slot that inherited an oldcatalog_xminstill has a youngrestart_lsn, so the WAL limit never triggers and the horizon has no bound.
Decision: retired, and must not be re-enabled anywhere, including Dedicated and Self-Managed once Siphon ships there. The mechanism is pure PostgreSQL and would happen the same way.
The fallback concept survived: the 2026-08-27 staging test proved that Siphon can fail over to a backup slot without data loss. What had to change was how the backup slot is kept current. It must be consumed, advanced, or recreated from a clean floor, never just rotated alongside the slots before it.
Approach 2: permanent slots advanced with pg_replication_slot_advance ❌
Section titled “Approach 2: permanent slots advanced with pg_replication_slot_advance ❌”What it was (Sep 2026): permanent slots advanced every hour to fixed
WAL-byte offsets behind the primary slot’s confirmed_flush_lsn. It was
stateless, only ever moved forward, and never dropped a slot. It was validated
on all three staging clusters.
Why it failed on gprd sec: the production#22859 window ran on 2026-09-11. The controlled tests in production#22939 ran on 2026-09-15.
pg_replication_slot_advance moves confirmed_flush_lsn to the requested
target. But restart_lsn, which catalog_xmin is tied to, can only advance to
a restart point recorded at a RUNNING_XACTS record.
| Test 1 | Test 2 | |
|---|---|---|
Bytes scanned (restart_lsn delta) | 515 MB | 4,041 MB |
| Elapsed | 20.6 s | 2 m 30 s |
restart_behind_confirmed after | 4.2 GB | 30 GB |
catalog_xmin age | rose | rose |
The gap grew instead of closing.
- Throughput view (RCA):
restart_lsnadvanced at ~1.6 GB/min against sec’s ~2.5 GB/min of WAL, so it lost ground every run. - Mechanism (source review): an advance call decodes up to the target and
confirms only once, at the end.
- While decoding, the slot keeps only the first candidate
restart_lsn/catalog_xminpast its previousconfirmed_flush_lsn, and discards the rest (LogicalIncreaseRestartDecodingForSlot/LogicalIncreaseXminForSlot). - So after each call,
restart_lsnlands just past the previous confirmed position, however far the target moved. - Test 2 decoded ~34 GB in 2m 30s, about six times sec’s WAL rate. Decode
throughput was not the binding limit, yet
restart_lsnmoved only about 4 GB.
- While decoding, the slot keeps only the first candidate
Either way, keeping the horizon tight would need a tight loop of small advances, each re-decoding the WAL behind it: a home-made consumer with worse semantics than a real one.
Staging passed because its WAL rate is 50–75× lower, so it could not hit this limit. The lesson for testing: validate anything that scales with volume against production-like WAL rates.
Impact: none. datfrozenxid stayed at 10.0% of the wraparound budget
throughout. Both tests were time-boxed, with an abort plan agreed in advance.
Rollback was clean: dropping the slot took catalog_xmin from ~1.30M back to
~19K within seconds.
Approach considered but not built: a real second consumer
Section titled “Approach considered but not built: a real second consumer”Siphon would run a second consumer against the backup slot, deliberately
trailing the primary by a set lag. A streaming consumer acknowledges
continuously, so catalog_xmin and restart_lsn would advance continuously,
and the failure mode would be designed out.
This was ruled out for now on cost and scope. It doubles the consumers per cluster (a second full logical decode of the WAL stream), and the lag has to be enforced through the consumer’s feedback, which is Siphon development work. Platform Insights judged it feasible but said it would need changes to the null-producer implementation and a lag mechanism based on WAL volume. It stays an option if the refresh design proves insufficient.
Open gaps
Section titled “Open gaps”Patroni failover orphans the backup slot
Section titled “Patroni failover orphans the backup slot”This has already happened on gstg: the main and ci backup slots disappeared after leader changes on 2026-10-27 and have not come back.
The backup slot has failover = true, so it can survive a leader change if it
was sync-ready on the promoted standby. The manually deployed cron does not
move with the leader. After a failover:
- the old leader’s script exits at its leader check and stops refreshing;
- nothing refreshes the slot on the new leader, so it holds WAL and
catalog_xminthere indefinitely. This is the INC-13574 failure mode with one slot instead of four; - any Alertmanager silence pinned to the old
fqdnno longer matches, so the Siphon lag alert pages for the new node.
On a hand-deployed cluster, a Patroni failover needs manual action (from production#23426):
- On the old leader, as
gitlab-psql, comment out the cron entry. - On the old leader, run the guarded drop in After a switchover or failover.
- Find the new leader:
sudo gitlab-patronictl list. - On the new leader, check that the backup slot has
active = f, then drop it. - Verify that WAL retention and
catalog_xminrecover on the new leader. - Expire the Alertmanager silence, or tell the SRE on-call if the alert has
already paged for the new
fqdn.
Then redeploy the script and cron on the new leader, or leave the cluster without a backup slot until you do.
The permanent fix is gitlab-patroni!214. It installs the script and the same cron on every node, and the leader check turns the standbys into no-ops. After a leader change, the new leader’s next run (within the hour) either refreshes the synced copy of the backup slot, or, if that copy wasn’t sync-ready and the slot is missing, recreates it. What still needs a person on a Chef-managed cluster:
- the old-leader cleanup (see Cleaning up the old leader is manual);
- any
fqdn-pinned Alertmanager silence, until runbooks!12101 merges.
The related corrective action is production-engineering#29661, “Prevent orphaned siphon backup replication slots from pinning vacuum” (Verification, due 2026-11-23).
Patroni and the Siphon slots
Section titled “Patroni and the Siphon slots”Patroni 3.3.4 drops any replication slot that is not in its slots: config or
ignore_slots, on primaries and replicas, whenever it re-reads the slot list:
when Patroni starts, after Postgres restarts (including a demotion), and when a
paused cluster is resumed. That is how gstg sec lost both Siphon slots during a
switchover on 2026-10-13
(production#23172).
- gstg: fixed. The sec, ci and main v18 roles set
bootstrap.dcs.ignore_slotsto{type: logical, plugin: pgoutput, database: gitlabhq_production}(chef-repo!8244). It has to be in the role: the cookbook rewrites the DCS config frombootstrap.dcson every chef run, so apatronictl edit-configalone is reverted. - gprd: fixed on 2026-11-16.
production#23458
(~C2) with
chef-repo!8309
(merged 2026-11-16 09:09 UTC) added the same matcher to
gprd-base-db-patroni-sec-v18,gprd-base-db-patroni-ci-v17(its c4 sub-roles inherit it) andgprd-base-db-patroni-main-v17, and it is in DCS on all three clusters. The Siphon slotsprd_sec_siphon_slot_1,prd_ci_siphon_slot_1andprd_main_siphon_slot_1are no longer dropped by a Patroni restart, switchover, pause/resume or failover. The flip side: on these clusters too, Patroni no longer cleans up the old leader’s stale copies, so the old-leader cleanup is needed after every gprd leader change.
What we’ve learned from rolling it out:
- One matcher covers every Siphon slot. Patroni matches exact values, so
{type: logical, plugin: pgoutput, database: gitlabhq_production}protects the primary slot, the backup slot and any futurepgoutputslot in that database, with no list of slot names to keep up to date. - Only the leader’s chef run applies it. The cookbook runs
patronictl edit-config --force --replacefrombootstrap.dcson the leader. A replica’s chef run doesn’t touch DCS. The setting is dynamic: Patroni applies it on its next HA loop, with no restart or switchover. - Verify it with
sudo gitlab-patronictl show-config | grep -A4 "^ignore_slots"on the leader, and check that no member logsDropped unknown replication slotorException when changing replication slotsin/var/log/gitlab/patroni/patroni.logafterwards. - Trade-off: Patroni no longer cleans up orphaned
pgoutputslots ingitlabhq_production. A retired Siphon slot has to be dropped by its owner, and stalesynced = fcopies need the guarded drop after every leader change and after building a replica from a leader snapshot. - New roles need it too. When ci and main move to v18 roles, those roles
need the same matcher. Restore clones that Patroni doesn’t manage (for
example the
postgres-main-v17-pitr-*nodes), and the archive and delayed clusters, are out of scope. - Rollback is a revert plus a chef run on the leader, or
printf 'ignore_slots: null\n' | sudo gitlab-patronictl edit-config --apply - --force. Don’t roll back with a restart, switchover or pause/resume planned on that cluster: the Siphon slots are exposed again as soon as the matcher is gone.
Cleaning up the old leader is manual
Section titled “Cleaning up the old leader is manual”After a switchover or failover
needs a person. An unplanned failover, including a watchdog reset, has no one
to run it. The candidate fix from production#23172 is to use Patroni’s
on_role_change callback (today /bin/echo on_role_change-do_nothing) to run
the guarded drop when a node becomes a replica, plus an alert on
exiting from slot synchronization in the Postgres logs. This must cover every
leader change, not only watchdog-driven ones. It is needed before the watchdog
is enforced in gprd, and it is a Phase 2 gate of the gprd sec watchdog change
(production#23457).
Until then, the gprd watchdog change requests run the guarded drop by hand
after each planned switchover.
Not yet packaged in Chef
Section titled “Not yet packaged in Chef”Until gitlab-patroni!214 merges and a cluster’s role runs the recipe, the
script is deployed by hand to the leader. That means no idempotent deployment,
no failover safety and no log rotation; /tmp/siphon_slot_backup_refresh.log
grows without limit. See Script for what the recipe changes.
Rollout order: gstg sec
(chef-repo!8338),
then gstg ci and main, then gprd, each after ignore_slots is live on that
cluster. The recipe needs gitlab-patroni 1.17.0. On 2026-11-13,
environments/gstg.json and environments/gprd.json in chef-repo pinned
gitlab-patroni at = 1.13.0, so check how the new version reaches the nodes
before merging a role change: a role that lists a recipe its pinned cookbook
doesn’t have fails its chef run. A pin bump also brings in every cookbook
change in between.
Planned follow-ups on the script: retry the create within the same run, and
make CREATE_TIMEOUT settable per cluster. The script reports only through
its log; there is no metric for “last successful refresh” or “backup slot
missing” yet, so a cluster that keeps skipping or failing is only visible in
the log and through the horizon alerts, once
those exist.
The retired rotation script is still in the cookbook
Section titled “The retired rotation script is still in the cookbook”files/default/manage_siphon_slots.sh
is still on gitlab-patroni master. It implements approach 1: every hour it
creates siphon_backup_slot_<HH> before dropping the slot from
RETENTION_HOURS ago (default 6). gitlab-patroni!214 does not remove it. It
should be deleted before someone finds it and wires it up.
No alerting on the horizon
Section titled “No alerting on the horizon”runbooks!11389 added the dashboard, but none of these alerts exists yet. They are tracked in production-engineering#30301:
catalog_xminage per slot, across all slots, including physical ones;age(datfrozenxid)rising whilepg_txid_xmin_agestays flat, which is the signature of this failure;- the age of inactive or orphaned slots.
The existing alerts don’t cover this:
PatroniXminAgeTooLarge(pg_txid_xmin_age) can’t see a slot pinningcatalog_xmin.- The
pg_xid_wraparoundandpg_txid_vacuum_to_wraparoundsaturation alerts fire far too late.
A related enhancement is under discussion: an S1 alert on the primary Siphon slot, so that the on-call engineer is engaged the moment it fails rather than finding out from downstream staleness.
PG18 (sec) also has idle_replication_slot_timeout, which is off by default.
The backup slot is always idle, so if this setting is ever enabled it must be
well above the refresh interval. Set long, it would be a built-in backstop for
orphaned slots. It also applies to the primary Siphon slot if Siphon is
disconnected for that long. PG17 (main, ci) doesn’t have it.
False-positive lag alerts on the backup slot
Section titled “False-positive lag alerts on the backup slot”SiphonLogicalReplicationSlotLagHigh (s2, pages Platform Insights) is defined
on pg_replication_slots_confirmed_flush_lsn_bytes{slot_name=~".*siphon.*"}
above 100 GiB for 5 minutes, on the primary only. It fires for each slot and
can’t tell the slot in use from the backup slot. The backup slot is never
consumed, so its lag grows until the next refresh, past 100 GiB on busy
clusters. On gprd sec it fired for about 23 hours in total over 7 days, with a
peak of 244 GiB.
The rule has no type label, so a silence can only match on slot_name
and fqdn. An fqdn-pinned silence stops matching after a failover.
Fix in progress:
runbooks!12101
(open) excludes slot_name=~".*siphon_backup.*" from the rule in gprd and
gstg, so backup slots stop alerting and every other Siphon slot keeps the same
threshold and routing. WAL retained by the backup slot is then covered only by
the disk saturation alerts and the
horizon alerts, once those exist.
The longer-term option is siphon#284 (open). Siphon 0.0.157-beta exports
siphon_replication_slot_lag_bytes{app_id, slot_name} for the slot it is
actually reading, and moving the alert to that metric would alert on whichever
slot Siphon uses.
Major PostgreSQL version upgrades
Section titled “Major PostgreSQL version upgrades”The PG17→PG18 upgrade of sec was the original motivation for this work. On
gstg sec (2026-06-18,
production#22298,
root cause in
dbo-issue-tracker#749),
the steps were:
- The Siphon slot’s on-disk state (
pg_replslot/<slot>/, withfailoverandsyncedset) had been captured in GCP disk snapshots. - A node built from those snapshots came up still holding
stg_sec_siphon_slot_1withsynced = t. - PostgreSQL refuses to drop a synced slot while the node is in recovery.
- A cascading standby cannot run slot sync to clean it up.
- So the slot couldn’t be removed there, which blocked
pg_upgrade.
The same workaround was then used for gprd sec in production#22366.
Backup slots reduce the blast radius, but they don’t make major upgrades hands-off. The complementary work is Siphon-side operational controls (pause/resume replication, clearing the LSN store), tracked in siphon#264.
Fallback: full rebuild when all slots are lost
Section titled “Fallback: full rebuild when all slots are lost”If the primary slot and the backup slot are both gone or invalidated, follow the Siphon service runbook: recreate the slot and run consistency checks with corrections for the affected database. A full re-snapshot (1–2 weeks on production) is the last resort.
The Orbit cutover plan, confirmed with the Orbit team:
- Orbit keeps serving from its current ClickHouse instance throughout.
- The datalake is rebuilt in parallel through a full Siphon refresh.
- Orbit reindexes into the new instance and cuts over, which takes about 1.5–2 hours.
So the customer impact is stale data, not an outage.
Related work:
- Consistency checks (shipped): siphon#276, “Source↔target completeness reconciliation (true-up & audit) for Siphon CDC”, closed 2026-10-19. The consistency checker compares PostgreSQL and ClickHouse range by range, and with corrections enabled re-reads the ranges that differ. This is what repairs the gap after a slot loss.
- Parallel snapshot plus Siphon extraction on a large VM with Hyperdisk Extreme, to cut full-rebuild time. Needs a POC.
- Narrowing the replicated data set so that a re-snapshot is smaller. Feasibility not yet assessed.
GitLab Dedicated (TBD)
Section titled “GitLab Dedicated (TBD)”Self-Managed (TBD)
Section titled “Self-Managed (TBD)”Related links
Section titled “Related links”- Siphon service runbook
- SiphonLogicalReplicationSlotLagHigh alert playbook
- SiphonEmergencyModeActive alert playbook
- Patroni service runbook
- PostgreSQL vacuum runbook
- XID wraparound alert
- Build cluster from snapshot
- Data Insights Platform overview
- NATS runbook
Issues, change requests and incidents
Section titled “Issues, change requests and incidents”| Reference | What |
|---|---|
| gitlab-org/gitlab#605537 | Design discussion and decision record for all three approaches |
| production#22807 / #22817 | INC-13566 / INC-13574, the incidents that retired approach 1 |
| production#22812 (confidential) | Incident review |
| production#22859 | Advance-design production window (approach 2) |
| production#22939 | Single-slot advance tests, which retired approach 2 |
| production#22298, #22366 | sec PG18 upgrades (synced-slot blocker) |
| production#23006 | First gprd sec refresh validation (2026-10-14) |
| production#23400 | gprd sec week-long refresh validation |
| production#23426 | gprd ci 24h refresh validation |
| gitlab-org/gitlab#605537 note_4198362282 | gprd sec and ci validation results, and the decision to use the gated refresh |
| production#23172, chef-repo!8244 | gstg ignore_slots, and the old-leader cleanup (step 3c) |
| production#23458, chef-repo!8309 | gprd ignore_slots on sec, ci and main |
| production#23352, #23457 | Patroni watchdog: gstg enforcement and the gprd sec rollout (switchover steps include the old-leader cleanup) |
| production-engineering#30300 | Tracking issue for this runbook |
| production-engineering#30301 | Alerting on slots pinning the catalog horizon |
| production-engineering#30302 | gstg backup slots missing or not refreshing |
| production-engineering#29661 | Orphaned backup slot lifecycle |
| gitlab-patroni!214 | Chef packaging of the refresh script |
| runbooks!11389 | Slot xmin age dashboard |
| runbooks!12101 | Exclude backup slots from SiphonLogicalReplicationSlotLagHigh |
| siphon#264 | Siphon API for pause/resume/clear LSN store |
| siphon#265 | Slot priority list (shipped) |
| siphon#276 | Source↔target completeness reconciliation (shipped) |
| siphon#284 | Adjust WAL alerts to account for backup slots |
Escalation
Section titled “Escalation”- Backup slot, refresh job, Patroni, WAL, vacuum horizon:
#g_database_operations(Database Operations). - Siphon behaviour, slot priority list, downstream data integrity:
#g_analytics_platform_insights(Platform Insights).