GKGPipelineInvariantViolated
The query compiler reached a state that upstream validation should have prevented; the generated SQL could have been incorrect. Severity s3, fires after 1m.
Overview
Section titled “Overview”The compiler rejected a query in a late stage (lowering, enforcement,
codegen or pipeline) that earlier validation should have made
unreachable. Each rejection is a compiler bug surfacing, and the same broken
state could in another variant produce incorrect SQL instead of a rejection.
Any non-zero count is anomalous.
Services
Section titled “Services”- Orbit service overview
- Owner team:
context_systems
Metrics
Section titled “Metrics”The firing expression, on gkg_query_engine_compiler_rejected_total (emitted by the service, see the
metrics catalog):
sum by (env, environment, stage) (rate(gkg_query_engine_compiler_rejected_total{failure_reason=~"lowering|enforcement|codegen|pipeline"}[5m])) > 0Alert Behavior
Section titled “Alert Behavior”- Silence via Alertmanager filtered on the alertname.
- Expected to never fire; each rejection is a compiler bug surfacing.
Severities
Section titled “Severities”- s3 during burn-in, promotion to s2 is planned alongside GKGAuthFilterMissing. The compiler rejected the query, so no incorrect SQL executed, but another variant of the same state might not be caught.
Verification
Section titled “Verification”Split the rejections by stage to see which invariant broke:
sum by (failure_reason) (rate(gkg_query_engine_compiler_rejected_total{failure_reason=~"lowering|enforcement|codegen|pipeline"}[5m]))Find the rejected query in the webserver logs (mode: webserver) around the
alert window; the rejection log carries the failing query shape.
Recent changes
Section titled “Recent changes”- argocd-apps MRs touching
services/gkg(a merge is a deploy). - Production issues labeled Service::Orbit.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Cause | Action |
|---|---|---|
| Rejections started with a deploy | New compiler bug in the release | Roll back the image tag; file an issue with the failing query |
| No deploy correlation | Latent bug triggered by a new query shape | Capture the query from the logs; the fuzzers usually reproduce the class |
Possible Resolutions
Section titled “Possible Resolutions”There is no operational fix: this is a code bug in the query compiler. If the
rejections started with a deploy, roll back the image tag. Capture the failing
query from the logs and file an issue with it; the fuzzers
(mise run fuzz:compile) can usually reproduce the class of failure.
Dependencies
Section titled “Dependencies”- None external: this is a query-compiler code bug by definition.
Escalation
Section titled “Escalation”Escalate to context_systems (#f_orbit_dev) with the failing query
attached. Treat a sustained rate as a rollback trigger, not something to
wait out.
Definitions
Section titled “Definitions”- Alert rule (hand-maintained; keep the expr above in sync when tuning it).