Skip to content
Runbooks

SiphonEmergencyModeActive

Overview

Section titled “Overview”

Fires when a Siphon producer has had emergency mode active continuously for more than one hour.

Emergency mode is a fallback write path: the producer stops publishing to NATS JetStream and writes replication events to object storage instead. Nothing is lost, the events are durably stored and can be replayed. But while it’s on:

  • The ClickHouse consumers receive no events, so every table fed by the affected producer stops advancing. Users querying that data see stale results.
  • An object storage backlog accumulates. The longer emergency mode stays on, the longer the drain afterwards.

It’s meant to be a short, deliberate mitigation, usually while NATS is unhealthy. An hour is long enough that a transient flip during a NATS blip or a producer deploy won’t page. If it’s still on after an hour, either NATS hasn’t recovered or the toggle was left enabled.

The PostgreSQL replication slot keeps advancing in emergency mode (the LSN is still acked), so this alert does not imply WAL accumulation on the Patroni primary. If WAL is also accumulating, expect SiphonLogicalReplicationSlotLagHigh to fire too, and treat that as the more urgent of the two.

Services

Section titled “Services”

Metrics

Section titled “Metrics”
  • siphon_failover_mode_active: boolean gauge, 1 while emergency mode is active.
  • Source: Siphon producer pods in the siphon namespace on the orbit-* clusters, scraped into the analytics-eventsdot Mimir tenant.
  • Aggregated with max by (environment, app_id, cluster), so we get one alert per producer rather than one per pod.
  • Siphon’s metrics in this tenant carry env="orbit-prd" / env="orbit-stg" rather than the gprd / gstg the rest of the fleet uses, and the alertmanager route feeding incident.io matches env=~"gprd|ops|thanos|production". The expression therefore derives a fleet-standard env from cluster via label_replace. Without it this alert would only reach Slack despite being an s2 with pager: pagerduty. Keep the orbit-(prd|stg) regex in step with the clusters we run, a cluster outside it won’t page.
  • Threshold: == 1 sustained for 1h.
  • Normal behaviour: flat 0. Any sustained 1 is a deliberate fallback state, not noise.

Alert Behavior

Section titled “Alert Behavior”
  • Clears automatically once emergency mode is off and the gauge returns to 0.
  • for: 1h requires the condition to hold continuously. A producer restart resets the pending timer, so a producer that keeps restarting while in emergency mode may never fire. Check the metric directly if you suspect that.
  • Silence only for the duration of a Change Request that deliberately keeps a producer in emergency mode (e.g. planned NATS maintenance), not indefinitely.
  • Should be rare. Emergency mode is not part of steady-state operation.
  • Previous incidents tagged with this alert

Severities

Section titled “Severities”
  • Default: s2, pager: pagerduty. It pages because the failure is invisible downstream, ClickHouse looks healthy, it’s just not receiving anything, so it needs an active push rather than a channel notification.
  • Escalate to s1 if NATS is unhealthy with no sign of recovery, if SiphonLogicalReplicationSlotLagHigh is firing on the same cluster (WAL accumulation on a production primary is the bigger risk), or if the backlog is large enough that the drain will exceed the freshness expectations of the downstream data.

Verification

Section titled “Verification”

Which producers are affected:

max(siphon_failover_mode_active{cluster=~"orbit-.*",namespace="siphon"}) by (environment, app_id)

How long each has been in emergency mode (seconds, over the last 24h):

sum_over_time(
  max(siphon_failover_mode_active{cluster=~"orbit-.*",namespace="siphon"}) by (environment, app_id)[24h:1m]
) * 60

Recent changes

Section titled “Recent changes”
  • Recent Siphon changes
  • Producer config and deployments live on the orbit-prd / orbit-stg clusters in the siphon namespace. Roll back via the usual Kubernetes rollback.

Troubleshooting

Section titled “Troubleshooting”

  1. Identify the producer. The app_id label names it.

  2. Check NATS JetStream health first. Turning emergency mode off before NATS is fixed will just flip it back on. See the NATS runbook.

  3. Check the producer pods. The logs should say why emergency mode engaged and whether the producer has been retrying NATS.

    Terminal window
    glsh kube use-cluster orbit-prd --no-proxy
    kubectl get pods -n siphon -o wide
    kubectl logs -n siphon -l app=postgres-producer-<db> --tail=200

  4. Size the object storage backlog before turning emergency mode off. A large backlog replayed all at once can overwhelm NATS and push the producer straight back into emergency mode.

  5. Once NATS is healthy, disable emergency mode and drain the backlog. Watch the publish rate and NATS health while it replays, and be ready to re-enable emergency mode if NATS starts to struggle.

  6. Verify the downstream caught up. Check that the ClickHouse tables fed by this producer have advanced past the gap and that siphon_failover_mode_active is back to 0.

Possible Resolutions

Section titled “Possible Resolutions”

Dependencies

Section titled “Dependencies”
  • NATS JetStream: the primary write path emergency mode bypasses.
  • Object storage: the fallback write path holding the backlog.
  • Siphon producer (postgres-producer-* pods in the siphon namespace).
  • ClickHouse consumers: starved of events while emergency mode is on.

Escalation

Section titled “Escalation”
  • Primary contact: #g_analytics_platform_insights. Most context: @ahegyi, @arun.sori, @ankitbhatnagar.
  • If NATS is the underlying problem and there’s no response from the primary contacts, escalate through the usual EOC path.

Definitions

Section titled “Definitions”
Section titled “Related Links”