Skip to content
Runbooks

Memorystore Redis TraceChunks Service

Logging

Section titled “Logging”

Summary

Section titled “Summary”

memorystore-redis-tracechunks is the GCP-managed MemoryStore for Redis instance (tracechunks-redis) that stores CI job trace (log) output in chunks. Rails writes trace chunks here as CI jobs run and reads them back when rendering job logs. If this instance degrades, CI job output may be delayed in becoming visible, or in severe cases lost.

It is a managed MemoryStore instance, not a Sentinel-based Redis VM.

Architecture

Section titled “Architecture”

The instance runs in us-east1 (gprd project gitlab-production, gstg project gitlab-staging-1). Rails reaches it through the trace_chunks storage selector; see the metrics-catalog definition metrics-catalog/services/memorystore-redis-tracechunks.jsonnet.

For connecting to the instance with redis-cli, see connecting-to-memorystore-redis.md.

Monitoring/Alerting

Section titled “Monitoring/Alerting”

The service has two SLIs, each from a different metric source:

  • primary_server: operations on the Redis primary, from the GCP Stackdriver exporter (stackdriver_redis_instance_redis_googleapis_com_commands_calls). Reflects server-side activity on the managed instance.
  • rails_redis_client: all TraceChunks operations issued from the Rails codebase (gitlab_redis_client_requests_total{storage="trace_chunks"}). Reflects client-side activity.

Because the two SLIs have independent sources, comparing them is the fastest way to tell a real traffic stop from a metric-pipeline break: Rails traffic present while primary_server is absent points at the Stackdriver exporter, not Redis.

Alerts

Section titled “Alerts”
Section titled “Links to further Documentation”