YugabyteDB Aeon — Runbook

Use this when something is wrong or an alert fired. Sections are ordered for an incident: triage first, then alerts, troubleshooting, and escalation. The Reference section at the bottom covers architecture, access, Vault paths, and how to make changes.

Managed-service reminder: Aeon is fully managed. We fix what we control via Terraform/console (sizing, disk, allow-lists, connections, backups/restore); node repair and internal cluster faults are Yugabyte’s — escalate those (see Incident response & escalation).

1. Quick triage (start here)

Scope: which cluster? prod or staging? one or many?
Health: Aeon Console → cluster → overall state Healthy? all nodes Live?
Client vs. DB: can other clients connect? If only one app is failing, suspect network/auth (→ Troubleshooting), not the DB.
Recent change: check the env’s recent Atlantis applies (plan/apply output is posted on the config-mgmt merge requests) — did a config change trigger this? If so, assess a revert.
Capture: screenshots (console health/Performance), exact error text, affected endpoint, timestamps — you’ll need these for escalation.

Then jump to the matching section: Alert playbook · Troubleshooting · Incident response · Backups & restore · Disaster recovery.

2. Alert playbook

Response for each Aeon native alert. These are the alerts Aeon exposes under Console → Alerts; severity is as configured. See Yugabyte’s alerting docs for the authoritative list and definitions (docs).

NOTE: On a single-zone fault, ZONE fault tolerance masks it in two stages. Leader failover to healthy nodes happens in ~3 seconds or less, so reads and writes should recover almost immediately. If the node stays down, Aeon only starts re-replicating the lost data onto other nodes after follower_unavailable_considered_failed_sec — 15 minutes (900s) by default (yb-tserver reference). So: a brief blip that clears in seconds is normal; if the cluster is still degraded (or the fault spreads) past ~15 minutes without re-replication kicking in, stop waiting and escalate.

Where to find the metrics

Navigate to https://cloud.yugabyte.com (log in with your Google SSO) → select the cluster, then Performance → Metrics tab (docs).

Time ranges: past 1h / 6h / 12h / 24h / 7d. Granularity: by region or individual node. Use Add Charts to pick metrics, Export to send to third-party monitoring tools.
Nodes tab → per-node status, zone placement, up/down state.
Backups tab → schedule state and last-run result.
Performance Advisor → slow / problematic queries.

Map the alert to the metric category so you know where to look:

Category	Concrete metrics	Relevant sections
YSQL ops (stg · prd)	ops/sec (`SELECT`/`INSERT`/`UPDATE`/`DELETE`), avg latency (p99), connection counts (standard + pooling), client connection wait time	2.4, 3.3, 3.4
Infrastructure (stg · prd)	CPU %, memory (GB), disk used vs. provisioned (GB), disk I/O bytes/sec + IOPS, network	2.2, 2.3
YB-TServer (stg · prd) / DocDB (stg · prd)	WAL, compaction activity, RPC queue sizes, RocksDB/SST metrics	2.1 (tablet peers), 2.3 (compaction overload)

2.1 Node / availability

Alert	Sev	Action
Majority of all primary nodes down	SEVERE	Cluster likely down. Page (see Incident response). Check console health. Do not apply Terraform to “fix” it. If no self-heal within minutes → Yugabyte Support, top severity.
Some of all primary nodes down	WARNING	ZONE FT should absorb a single-zone loss. Watch for auto-heal. If it persists or spreads → treat as the SEVERE case.
More than 66% of read replica nodes down	SEVERE	Read-replica capacity degraded (reads may fail/route to primaries). Check console; if no self-heal → Yugabyte Support.
More than 34% of read replica nodes down	WARNING	Monitor; confirm reads still served. Ticket if no recovery.
Percentage of tablet peers is high	WARNING	Usually under-replication/rebalancing after a node change or scale. Normally transient (watch Performance → tablet-peer metrics). If it doesn’t clear → Yugabyte Support.

2.2 Storage

Alert	Sev	Action
Node Free Storage below 25%	SEVERE	Increase `disk_size_gb` now via MR (see Making changes). Near-full disks cause write stalls. Then investigate growth with the owning team.
Node Free Storage below 40%	WARNING	Plan a disk increase; investigate growth (bloat, retention, unexpected load).
Node disk throughput utilization exceeded 100%	WARNING	IO saturated → latency. Find the workload (Performance / slow queries, 3.3); scale `num_cores`/`num_nodes`.
Node disk throughput utilization exceeded 80%	WARNING	Trending hot; right-size capacity before it saturates.
Node disk IOPS utilization exceeded 100% / 80%	WARNING	Same as throughput — IO-bound; investigate workload and scale.

2.3 Compute / memory

Alert	Sev	Action
CPU Utilization Exceeded 90%	SEVERE	Check Performance/slow queries (3.3). Scale `num_cores`; engage owning team on query load.
CPU Utilization Exceeded 70%	WARNING	Capacity-plan; investigate if sustained.
Memory Utilization Exceeded 90%	SEVERE	Investigate query/workload memory; scale up. If unexplained/leaking → Yugabyte Support.
Memory Utilization Exceeded 70%	WARNING	Monitor; correlate with workload.
Cluster Queues Overflow and/or Compaction Overload	SEVERE	Write/compaction overload. Reduce write load if possible; scale; if it won’t drain → Yugabyte Support.

2.4 Connections

Alert	Sev	Action
Exceeded 85% YSQL Connections Limit	SEVERE	Risk of “too many clients”. Check for a client-side leak; enable/right-size connection pooling (3.4).
Exceeded 85% YSQL Connection Pooling Server/Client Connections Limit	SEVERE	Pooler near capacity; right-size pool / client config.
Exceeded 60% YSQL Connections Limit	WARNING	Trending toward the limit; review pool sizing with the app team.
Exceeded 60% YSQL Connection Pooling Server/Client Connections Limit	WARNING	Same — pooler trending hot.

2.5 Billing

Not an SRE remediation — route to the account/billing owner (see Internal escalation).

Alert	Sev	Action
Invoice Payment Failure	SEVERE	Notify billing owner immediately — risk to the account/clusters.
Credit Card Expiring within 3 months	SEVERE	Notify billing owner to update payment method.
Credit Usage exceeds 90% limit	SEVERE	Notify billing owner; risk of hitting the cap.
Credit Usage exceeds 75% / 50% limit	WARNING	Inform billing owner for planning.

3. Troubleshooting

Most fixes below are applied via Terraform/Atlantis — see Making changes for how to open an MR and apply.

3.1 Can’t connect to the cluster

Symptoms: timeouts, “connection refused”, “no route to host” from the app or psql.

Diagnose:

Console → cluster Healthy, nodes Live? If not → 2.1.
Right endpoint? Private endpoint (PSC) for in-VPC clients; public endpoint only if public_access = true.
Source IP allowed? In-VPC clients reach the cluster over the PSC endpoint and need no allow-list entry. Public access needs the client IP/CIDR in ybm_allow_list.
Correct ports? YSQL 5433, YCQL 9042.

Fix:

Missing IP → add the CIDR to allow_list.cidr_list, MR → apply.
Wrong endpoint → check the private endpoint and validate it against the Runway PSC consumer connector. The connector is owned by the Runway team; ask in #f_runway for help.

3.2 Authentication failures

Symptoms: “password authentication failed”, “role does not exist”.

Confirm credentials against the input Vault secret (ci mount): ysql_username/ysql_password, ycql_* for YCQL (see Vault paths).
App reads creds from the Runway output secret (stg, prd) — confirm it matches the input secret.
Login credentials for manual connections are also in 1Password, read-only (iam_ops_ro) — in the Engineering vault, items IAM YB Prod Read_Only Role (prod) / IAM YB Staging Read_Only Role (staging).

3.3 High latency / slow queries

Symptoms: p99 latency spike, app timeouts, CPU/IO pegged.

Console → Performance: CPU, IOPS, connections, p99.
CPU > 80% or IO saturated → undersized (2.2/2.3); connections near max → pooling (3.4).
Use Aeon’s Performance Advisor / slow-query view to find offending queries.

Fix: scale num_cores/num_nodes in place (see Making changes); query tuning with the owning team (out of scope for infra).

3.4 Connection pool exhaustion

Symptoms: “too many clients”, “remaining connection slots reserved”.

Console → connections vs. limit. Usually a client-side leak or undersized pooler. Here “client” is our Go service and its (currently untuned) pgx pool.

Sizing the limit (~15 connections per vCPU core): Aeon’s YSQL connection limit scales with cluster size — roughly 15 connections per vCPU core across the cluster. So total connections ≈ 15 × num_cores × num_nodes. Use this to check whether the cap is a workload problem or just an undersized cluster:

e.g. a 3-node cluster at 4 vCPU each → ~15 × 4 × 3 = 180 connections.
If the app legitimately needs more, scale num_cores/num_nodes rather than fighting the limit; otherwise pool/right-size the client first.

3.5 Node down / cluster unhealthy

See 2.1. ZONE FT tolerates losing one zone; Aeon auto-heals. Don’t apply Terraform to fix a down node — it won’t help and may collide with healing. No recovery / multiple nodes → Yugabyte Support.

3.6 Terraform / Atlantis apply failures

Symptom	Cause / fix
”Credentials cannot be modified after creation”	Credentials are immutable after creation; revert that part of the change.
Aeon API error on `cloud_type`/`region`/`cidr`/`cluster_tier`/`cluster_type`	Immutable attribute — needs a new cluster + migration, not an edit.
Provider auth / long HTML error	Aeon API token expired/invalid; rotate and update the `ci`-mount Vault secret (see Vault paths).
“changes made outside of Terraform” (drift)	Someone edited in the console. Revert the console change or codify it in Terraform.
”cluster edit operation in progress”	A prior edit didn’t finish; the cluster must leave that state first. Check the console task list; retry once clear.

3.7 Connecting via Cloud Shell for troubleshooting

Cloud Shell is a browser-based SQL shell built into the console — no local client, VPN, or allow-list entry needed.

Finding it in the console:

Go to https://cloud.yugabyte.com and log in with Google SSO.
Select the right account/env (iam-prod or iam-staging) and open the cluster (iam-yugabyte-gprd / iam-yugabyte-gstg).
On the cluster page, click Connect (top-right).
In the dialog, choose Launch Cloud Shell.
Set Database to iam-service, User to iam_ops_ro, pick the YSQL API, then Confirm — enter the password when prompted.

Use the iam_ops_ro read-only user — it’s scoped for troubleshooting and won’t risk write changes. Its credentials are in 1Password, in the Engineering vault, items IAM YB Prod Read_Only Role (prod) / IAM YB Staging Read_Only Role (staging).

Cloud Shell sessions are time-limited (~1 hour) and rate-limited by Yugabyte; relaunch if the session expires.

4. Incident response & escalation

4.1 Severity

Severity	Example	Action
S1 / Critical	Prod cluster down/unreachable; data loss/corruption; restore needed	Page on-call; Yugabyte Support at top severity; declare incident
S2 / High	Severe degradation but serving; one feature broken	Page on-call; Yugabyte ticket if cluster-side
S3 / Moderate	Single node down but cluster healthy; disk > 80%	Investigate same business day; scale/clear; ticket if no self-heal
S4 / Low	Backup warning, minor drift, capacity planning	Routine MR / follow-up

Staging (gstg) issues are normally ≤ S3 unless they block a release.

Our S1–S4 map onto Yugabyte’s support Severity 1–4 — set the matching severity when filing a ticket. For our ZONE (availability-zone) fault-tolerant clusters, Yugabyte’s target initial response times are: Sev 1 — 1 hour, 24×7; Sev 2 — 4 business hours; Sev 3 — 8 business hours; Sev 4 — next business day (support policy). These are response, not resolution, targets; uptime/credit commitments are in the separate SLA.

4.2 First responder checklist

See Quick triage. Then mitigate within our control (allow-list, scaling, disk); escalate the rest.

4.3 Internal escalation

On-call / paging: [PLACEHOLDER]
Owning team for this DB: [PLACEHOLDER]
Billing owner: [PLACEHOLDER]
Incident channel / process: [PLACEHOLDER]

4.4 Yugabyte Support (vendor)

How to open: Yugabyte support portal at https://support.yugabyte.com/ (also reachable via Aeon Console → Support). (PLACEHOLDER: account/org, support tier & SLA, account manager / shared Slack)
Set ticket severity to match impact (prod down = their highest).
Include: cluster name + ID (cluster_id), region, time window, symptom, metrics screenshots, and what you’ve already checked.
Escalate for: nodes not self-healing, replication/tablet issues, managed backup/restore failures, suspected platform bugs, planned-maintenance impact.
Managing who can file tickets: to add/remove people from Yugabyte’s support tracker, see Managing Yugabyte support access.

4.5 What we handle vs. what Yugabyte handles

We handle (Terraform/console)	Yugabyte handles
Sizing/scaling, disk growth	Node provisioning & repair
Allow-lists, endpoints, network access	Underlying infra / availability
Backup policy, triggering restores	Backup execution, storage durability
Credentials at creation, app config	Managed-plane / control-plane bugs

5. Backups & restore

Backup policy is configured in Terraform. Verifying backups and restoring are operational actions, done here / in the Aeon console.

5.1 Verify backups

Aeon Console → cluster → Backups.
Latest scheduled backup SUCCEEDED and within the expected window?
Retention matches Terraform (no manual drift)?
A missed/failed backup should alert; persistent failures → Yugabyte Support.

5.2 On-demand backup

Before risky operations (major scaling, schema migrations): Console → Backups → Backup Now. Note its ID/timestamp.

5.3 Restore

Performed via the Aeon Console / API, not Terraform.

Options:

Restore in place — overwrites current data. Destructive; only to recover a known-good state after loss/corruption.
Restore to a new/another cluster — safer for partial recovery/investigation; bring up a target, restore into it, repoint the app.

Procedure (in-place):

Pause app writes if possible.
Console → cluster → Backups → select backup → Restore.
Wait for completion (scales with data size).
Validate data, resume traffic.
Record the action in the incident log.

5.4 Point-in-time / incremental

Point-in-time recovery (PITR) is not currently enabled on these clusters, so restores can only target a scheduled/on-demand backup, not an arbitrary point in time.

6. Disaster recovery

6.1 What’s protected

ZONE fault tolerance → survives loss of one availability zone automatically (no DR action; see 2.1).
Not protected: region loss or full cluster loss/deletion. Clusters are single-region (e.g. gstg us-east1).
Backstop = backups (see Backups & restore); recovery point bounded by backup interval.

6.2 Accidental cluster deletion / destroy — need to test

Declare an incident (see Incident response); stop further applies on the env.
Recreate from Terraform (env dir rebuilds ybm_vpc + ybm_cluster). Rebuilt cluster is empty.
Restore data from the latest good backup (5.3).
Reconcile Vault outputs (ca.crt/endpoints/passwords on the Runway secret); confirm the app picks up the new endpoint.
Validate, restore traffic.

6.3 DR readiness

Test restores periodically (5.3, restore-to-new-cluster) — an untested backup isn’t a backup. Record the measured RTO.
Keep Terraform able to recreate the cluster cleanly (no console drift, see 3.6).
Re-validate escalation contacts (4.3/4.4) each DR drill.

7. Reference: environments, access & making changes

Environments

Env	Aeon account	Cluster	Database
Staging (gstg)	`iam-staging`	`iam-yugabyte-gstg`	`iam-service`
Production (gprd)	`iam-prod`	`iam-yugabyte-gprd`	`iam-service`

Architecture: Aeon-managed YugabyteDB, single-region per cluster, ZONE fault tolerance. Clusters/accounts/databases per env — see the table above. YSQL on 5433, YCQL on 9042. We currently run 3 nodes across 3 zones.
Access: Aeon Console at https://cloud.yugabyte.com — log in with Google SSO. In-VPC clients connect over the Private Service Connect (PSC) endpoint; public access only when public_access = true and the client IP/CIDR is on the allow-list. Cloud Shell troubleshooting uses the read-only iam_ops_ro user against iam-service (see 3.7).

Vault paths

App/output creds (Runway mount): what the app reads — deep-links: iam-data-access staging · iam-data-access production.
Input creds/API token (ci mount): the TF cluster credentials (ysql_username/ysql_password, ycql_*, Aeon API token) live in the ops (ops-gitlab-net) vault, not the engineering vault — deep-links: production · staging.
1Password: login creds for manual connections, read-only (iam_ops_ro) — in the Engineering vault, items IAM YB Prod Read_Only Role (prod) / IAM YB Staging Read_Only Role (staging).

Making changes

Everything we control is Terraform via Atlantis — sizing (num_cores/num_nodes), disk_size_gb, ybm_allow_list, connection pooling, backup policy. Open an MR and atlantis apply — the owning team can self-serve; this is not gated to Infra.

Credentials, cloud_type, region, cidr, and cluster tier/type are immutable after creation. Attempting to change them fails the apply (see 3.6); an immutable change means a new cluster + migration. For the full set of immutable (and mutable) cluster settings, see Immutable & mutable cluster settings.
- The immutable “credentials” here are the Aeon user/password Terraform uses to apply changes.
Restores are done in the Console/API, not Terraform (see 5.3).
Terraform lives in the config-mgmt repo under environments/iam-yugabyte-gprd (prod) and environments/iam-yugabyte-gstg (staging).

Outstanding fill-ins (runbook)

2.5 / 4.3 Billing owner; on-call/paging, owning team, incident channel
4.4 Yugabyte support tier, portal account, SLA, account contact
5.4 Confirm PITR state