GKGAllowlistRejectedBurst
Sustained ontology-allowlist violations, above 5 per minute; schema drift or an enumeration attempt. Severity s3, fires after 5m.
Overview
Section titled “Overview”Queries are referencing entities, properties or relationships outside the ontology allowlist at more than 5 per minute. Two very different causes share this signature: a client built against a newer or older graph schema than the server exposes (drift), or a caller enumerating the schema surface to see what exists (probing).
Services
Section titled “Services”- Orbit service overview
- Owner team:
context_systems
Metrics
Section titled “Metrics”The firing expression, on gkg_query_engine_compiler_rejected_total (emitted by the service, see the
metrics catalog):
sum by (env, environment, stage) (rate(gkg_query_engine_compiler_rejected_total{failure_reason=~"ontology|ontology_internal"}[5m])) * 60 > 5Alert Behavior
Section titled “Alert Behavior”- Silence via Alertmanager filtered on the alertname.
- Drift rejects a small stable set of names repeatedly; enumeration walks many names once.
Severities
Section titled “Severities”- s3, no paging.
ontology_internalrejections deserve the most attention: no legitimate client references internal-only schema elements.
Verification
Section titled “Verification”sum by (failure_reason) (rate(gkg_query_engine_compiler_rejected_total{failure_reason=~"ontology|ontology_internal"}[5m])) * 60Pull the rejected names from the compiler rejection logs. Real schema drift
rejects a small stable set of names repeatedly; enumeration walks many names
once. ontology_internal rejections mean the query referenced internal-only
schema elements, which no legitimate client does.
Recent changes
Section titled “Recent changes”- argocd-apps MRs touching
services/gkg(a merge is a deploy). - Production issues labeled Service::Orbit.
Troubleshooting
Section titled “Troubleshooting”| Symptom | Cause | Action |
|---|---|---|
| Small stable set of names rejected repeatedly | Schema drift between client and server | Check for a recent SCHEMA_VERSION or ontology change; update or roll back; glab orbit remote schema shows what the server exposes |
| Many names each rejected once | Enumeration of the schema surface | Capture the caller identity; follow the security process |
ontology_internal rejections present | Caller referencing internal-only elements | Treat as probing |
Possible Resolutions
Section titled “Possible Resolutions”- Schema drift: check whether a
SCHEMA_VERSIONbump or ontology change shipped recently and whether the caller (glab, Duo tooling) needs a release to match;glab orbit remote schemashows what the server exposes. - Enumeration: rate-limiting lives in Rails; capture the caller identity from the logs before it stops.
Dependencies
Section titled “Dependencies”- The ontology allowlist shipped with the running GKG version.
- Clients built against the graph schema (glab, Duo tooling).
Escalation
Section titled “Escalation”Ask in #f_orbit_dev (context_systems). For suspected enumeration of
internal schema, follow the security process.
Definitions
Section titled “Definitions”- Alert rule (hand-maintained; keep the expr above in sync when tuning it).