GKGPipelinePostCompileErrorRateHigh
Post-compile failure rate is above 1%, isolating server-side reliability from client-input quality. Severity s3, fires after 5m.
Overview
Section titled “Overview”The failure rate of queries that passed compilation exceeds 1%. Excluding
compile_error filters out bad client input, so what remains (execution,
authorization, security, internal) is the server’s own reliability. This
mirrors the “Pipeline success rate (1h), post-compile” gauge on the overview
dashboard. When GKGQueryingErrorRateHigh fires but this one does not, the
problem is client input, not the service.
Services
Section titled “Services”- Orbit service overview
- Owner team:
context_systems
Metrics
Section titled “Metrics”The firing expression, on gkg_query_pipeline_queries_total (emitted by the service, see the
metrics catalog):
sum by (env, environment, stage) (rate(gkg_query_pipeline_queries_total{status!~"ok|compile_error"}[5m])) / sum by (env, environment, stage) (rate(gkg_query_pipeline_queries_total{status!="compile_error"}[5m])) > 0.01Alert Behavior
Section titled “Alert Behavior”- Silence via Alertmanager filtered on the alertname.
- Fires without GKGQueryingErrorRateHigh when the server is failing but total volume dilutes the aggregate rate.
Severities
Section titled “Severities”- s3, no paging. Excludes client input errors, so this is server-side reliability degrading.
Verification
Section titled “Verification”sum by (status) (rate(gkg_query_pipeline_queries_total{status!~"ok|compile_error"}[5m]))execution: ClickHouse failing queries, cross-checkGKGExecutionFailureRate.authorization: the Rails redaction/authorization exchange failing, cross-checkGKGAuthorizationFailureRate.security: cross-checkGKGSecurityRejected.
Recent changes
Section titled “Recent changes”- argocd-apps MRs touching
services/gkg(a merge is a deploy). - Production issues labeled Service::Orbit.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Cause | Action |
|---|---|---|
execution dominates | ClickHouse failing queries | Follow GKGExecutionFailureRate |
authorization dominates | Rails redaction exchange failing | Follow GKGAuthorizationFailureRate |
| No single mode dominates, rose with a deploy | Release regression | Roll back the image tag and investigate offline |
Possible Resolutions
Section titled “Possible Resolutions”Follow the dominant failure mode’s own playbook (linked above). If no single mode dominates and the rate rose with a deploy, roll back the image tag and investigate offline.
Dependencies
Section titled “Dependencies”- Graph ClickHouse (ClickHouse Cloud)
- GitLab Rails (JWT minting, internal Orbit API)
Escalation
Section titled “Escalation”Ask in #f_orbit_dev (context_systems).
Definitions
Section titled “Definitions”- Alert rule (hand-maintained; keep the expr above in sync when tuning it).