Skip to content
Runbooks

RunnerShardFallingBackFromJobRouter

Managers in a shard have FF_USE_JOB_ROUTER on but have polled GitLab directly for 15 minutes.

Overview

Section titled “Overview”

A manager with FF_USE_JOB_ROUTER on asks Rails where the router is (GET /api/v4/runners/router/discovery), opens a gRPC connection to Relay, and calls GetJob. When either of the first two steps fails, the manager sends that poll as POST /api/v4/jobs/request instead and counts a fallback with a reason:

  • no_discovery: discovery returned no router. Rails answers 501 when the job_router flag is off for the runner or Relay is disabled in the Rails configuration. A 403, a 5xx, or a connection error counts the same way. The runner caches the answer for five minutes, so one failed discovery keeps the manager off the router for at least that long.
  • dial_failed: discovery returned a router, but the manager could not open a connection to it.

Jobs keep running on the direct path. The alert exists because the shard left the router path without anyone deciding to take it off, and the Relay-side alerts cannot see traffic that never reaches Relay.

The other two reasons, breaker_tripped and breaker_open, mean the manager reached Relay and GetJob failed. The Job Router SLI alerts cover those.

Services

Section titled “Services”
  • ci-runners: the managers that fall back and expose the metric.
  • Relay (kas): the router path.
  • Rails: the discovery endpoint and the job_router flag.

Metrics

Section titled “Metrics”
  • gitlab_runner_job_router_fallbacks_total{reason}: fallbacks per manager, by reason. Runner-side Fallbacks/sec on the Job Router dashboard shows it.
  • gitlab_runner_job_router_discovery_cache_events_total{result}: discovery cache hits and misses.

The shard label names the shard.

Alert Behavior

Section titled “Alert Behavior”
  • Fires when a shard records any no_discovery or dial_failed fallback, as a 5-minute rate, for 15 minutes. Grouped by shard and reason.
  • Routes to the Runners Platform alert channel, #g_runners_platform_alerts, like the other runner manager alerts. It does not page.
  • During a deliberate rollback, with the Rails job_router flag off, every shard with the runner flag on fires with no_discovery. Silence the alert for the rollback window.

Severities

Section titled “Severities”

S4. No jobs are lost. The shard runs on the direct path until the cause is fixed.

Verification

Section titled “Verification”
  1. On the Job Router dashboard, Runner-side Fallbacks/sec by reason for the shard.
  2. For no_discovery, the Rails side of the discovery calls is in Kibana: discovery requests. 501 is the flag off or Relay disabled; 403 is the runner token; 5xx is Rails. The managers log Discovering Job Router... with the outcome, for example not configured/enabled for a 501, in pubsub-runner-inf-gprd-*.
  3. For dial_failed, the managers log Job router dial failed, falling back to direct job requests with the address they tried, in pubsub-runner-inf-gprd-*.

Recent changes

Section titled “Recent changes”
  • The Rails job_router flag: feature flag log.
  • A shard’s job_router_url changed or a shard was added in argocd/apps or chef-repo.
  • A change to the kas frontend on HAProxy main, to the kas-internal Private Service Connect endpoint, or to the firewall rules on the networking page.
  • A Relay deploy, if the kas frontend health check took every backend out.

Troubleshooting

Section titled “Troubleshooting”
  • no_discovery on every shard: the Rails flag is off or Relay is disabled in Rails. Check the flag first.
  • no_discovery on one shard: discovery fails for that shard’s token. Read the status in the Rails log above and the manager log.
  • dial_failed: the shard cannot reach its router address, kas-internal.<env>.gitlab.net:8150 through PSC for the Linux Kubernetes managers or kas.gitlab.com for the others. Check the shard’s job_router_url, then DNS and connectivity from a manager pod, then the kas frontend on HAProxy main.

Possible Resolutions

Section titled “Possible Resolutions”
  • Flag off by mistake: turn it on. Flag off on purpose: silence the alert for the rollback window.
  • Wrong job_router_url: fix the shard’s values file or role.
  • Network: restore the PSC endpoint or firewall rule. If the kas frontend has no healthy backend, that is a Relay incident; follow the Job Router SLI alerts.

Dependencies

Section titled “Dependencies”
  • The Rails discovery endpoint and the job_router flag.
  • The kas frontend on HAProxy main, reached through the kas-internal PSC endpoint or kas.gitlab.com.

Escalation

Section titled “Escalation”

Runners Platform runs the managers and their network path, and owns this alert. Reach them in #g_runners_platform. Runner Core owns the Job Router, the discovery endpoint, and the Rails flags. Reach them in #g_runner_core.

Definitions

Section titled “Definitions”
  • The rule is in libsonnet/alerts/runner-manager-alerts.libsonnet. Regenerate with ./scripts/generate-jsonnet-rules.sh mimir-rules-jsonnet/runners-platform-alerts.jsonnet.
  • Tune the threshold there: any fallback for 15 minutes.
Section titled “Related Links”