Skip to content
Runbooks

NATS for audit event streaming

This NATS cluster runs in the nats namespace of the gprd-gitlab-gke cluster (and gstg-gitlab-gke for staging). It is separate from the NATS service used by Orbit and Analytics, documented in NATS.

GitLab publishes audit events to the JetStream stream audit_events_streaming. The stream has 256 partition subjects, each with a durable consumer (audit_streaming_consumer_<n>), plus the instance lane consumer audit_streaming_consumer_instance. Sidekiq drains them: AuditEvents::Streaming::NatsConsumerWorker runs every minute and fans out one AuditEvents::Streaming::NatsPartitionConsumerWorker per partition, which sends the events to external streaming destinations.

The stream keeps messages for 24 hours (max_age). Messages that are still unprocessed after that are dropped and cannot be recovered.

Oldest pending message age high

Section titled “Oldest pending message age high”

Alert: AuditEventStreamingNatsOldestPendingMessageAgeHigh

What it means

Section titled “What it means”

A consumer’s oldest unacknowledged message was published more than 6 hours ago, and has been for 15 minutes. NATS exports no message timestamps, so the alert compares the consumer’s ack floor with the stream’s last sequence from 6 hours ago.

This catches two cases that the 30-minute AuditEventStreamingNatsConsumerSilentDropRisk alert misses, because the consumer is still acknowledging messages:

  • the consumer is draining, but more slowly than events arrive
  • one message keeps failing while newer messages are delivered, holding back the ack floor

There are about 18 hours left before the stream starts dropping the oldest messages.

Limits:

  • The instance lane consumer audit_streaming_consumer_instance is not covered. It has never acknowledged a message, so its ack floor is 0, and including it would make its first message look older than 6 hours.
  • A partition that receives nothing for 6 hours and then gets a message that fails for 15 minutes fires the alert, even though the message is new. On GitLab.com each partition gets a message every few seconds, so this does not happen in practice. Keep it in mind if the rule is copied to a quieter environment such as staging.

What to check

Section titled “What to check”

  1. Are the drainers running? In Kibana, use the pubsub-sidekiq-inf-gprd* index:

    json.class:"AuditEvents::Streaming::NatsPartitionConsumerWorker"

    Look for skipped runs (nats_unreachable, stream_not_found) and for runs that dispatch nothing. Also check that the ops flag audit_event_streaming_nats_consumer (the consumer kill switch) is enabled and that silent mode is off.

  2. Is one destination failing? Check the dispatch failure ratio and gitlab_audit_event_streaming_nats_dispatch_destination_error_total on the dashboard. A destination that fails for hours holds back its partitions.

  3. Is NATS reachable? See AuditEventStreamingNatsConsumerUnreachable.

With the redelivery flag audit_event_streaming_nats_consumer_redelivery enabled, a message that keeps failing stops being redelivered after about 6 hours. See Max deliveries reached.

Max deliveries reached

Section titled “Max deliveries reached”

Alert: AuditEventStreamingNatsMaxDeliveriesReached

What it means

Section titled “What it means”

With the ops flag audit_event_streaming_nats_consumer_redelivery enabled, each consumer delivers a message at most 10 times (max_deliver). Redeliveries back off over 2m, 5m, 15m, 30m and then every hour, so the last attempt happens about 6 hours after the first.

When the last allowed delivery fails, the drainer increments gitlab_audit_event_streaming_nats_max_deliveries_total and logs the message’s stream sequence. JetStream then stops redelivering it. The event was not delivered to its destination.

The message is still in the stream and can be read by its stream sequence until 24 hours after it was published. After that, the stream drops it.

The alert fires within a few minutes of the event and resolves 15 minutes later. Resolving does not mean the message was delivered.

1. Find the messages

Section titled “1. Find the messages”

The drainer logs one warning per message. In Kibana, use the pubsub-sidekiq-inf-gprd* index:

json.subcomponent:"application_json" AND json.message:"NATS audit event message failed on its last allowed delivery"
FieldMeaning
json.stream_sequenceStream sequence. Use it to read the message.
json.consumer, json.partitionDurable consumer and partition
json.num_deliveredDelivery count at the last attempt (10, or more for messages already past the limit when the flag was enabled)
json.outcomedestination_error (the destination rejected or could not be reached) or failed (a GitLab-side error)

The log entry also includes audit_event_id and group_id. Treat them as customer data: do not paste them into Slack, public issues or merge requests. Use a confidential issue.

2. Find and fix the cause

Section titled “2. Find and fix the cause”
  • destination_error: an external destination has been failing for about 6 hours. Check gitlab_audit_event_streaming_nats_dispatch_destination_error_total and the dispatch failure ratio on the dashboard. Further messages for that destination will keep reaching the limit until it recovers.
  • failed: a GitLab-side error during dispatch. Check Sentry for exceptions from AuditEvents::Streaming::NatsPartitionConsumerWorker and the AuditEventStreamingNatsDispatchFailureRatioHigh alert.

3. Check that the message is still readable

Section titled “3. Check that the message is still readable”

This step needs an SRE. NATS only accepts mutual TLS clients whose certificate maps to a known Rails identity, so it cannot be done from the Teleport Rails console VMs, which have no NATS configuration, or from a laptop. Do not port-forward to the NATS pods: it would need the production NATS client certificate outside the cluster, and puts message data on your machine.

Use a Rails console in a Sidekiq pod. Sidekiq’s NATS identity ([email protected]) is the only one with JetStream API access. The webservice identity can only publish events. Permissions are in the NATS gprd values.

  1. Find the Sidekiq shard that runs the drainers: in Kibana, check json.shard on recent AuditEvents::Streaming::NatsPartitionConsumerWorker jobs. Its pods are known to have working JetStream access. Sidekiq shards run under different service accounts, so another shard’s pods may not have the same NATS access.

  2. Pick a running pod from that shard’s deployment in the gitlab namespace on gprd-gitlab-gke, and start a Rails console in its sidekiq container.

  3. Check that the console has NATS configured:

    Gitlab::Nats.configured?

  4. Read the message’s metadata. End each line with ; nil so the console does not print the message. Do not print msg or msg.data: the body holds audit event data (actor, IP address, target).

    Use a separate client with Sidekiq’s inbox prefix (_INBOX_SK). A Rails console is not Sidekiq at runtime, even in a Sidekiq pod, so Gitlab::Nats.client would use the default _INBOX prefix, which the Sidekiq identity is not meant to keep. A separate client also leaves the shared one untouched.

    seq = <stream_sequence>
    stream = 'audit_events_streaming'
    msg = published_at = nil
    client = Gitlab::Nats::Client.new(
      **Gitlab::Nats.send(:connection_options).merge(inbox_prefix: Gitlab::Nats::SIDEKIQ_INBOX_PREFIX)
    ); nil
    

    # Below the stream's first sequence, the message has already been dropped.
    client.stream_info(stream)&.state&.first_seq
    

    # Raises NATS::JetStream::Error::NotFound if the message is gone.
    msg = client.send(:jetstream).get_msg(stream, seq: seq); nil
    published_at = Gitlab::Json.parse(msg.data)['published_at'] if msg; nil
    

    { seq: msg&.seq, subject: msg&.subject, bytes: msg&.data&.bytesize,
      published_at: published_at, readable_until: (Time.iso8601(published_at) + 24.hours if published_at) }
    

    client.close

    readable_until is the deadline to act. If published_at is missing, treat the message as close to its deadline. If get_msg raised, the summary shows seq: nil. Do not reuse values from an earlier run.

    Reading a message needs $JS.API.STREAM.MSG.GET.audit_events_streaming in the Sidekiq identity’s publish permissions. A missing permission does not raise a permission error: the request waits about 5 seconds and raises NATS::Timeout, and the client reconnects in the background. If stream_info works but get_msg times out, the permission is probably missing. A missing $JS.API.STREAM.INFO.audit_events_streaming makes stream_info time out the same way. The NATS server logs Publish Violation with the denied subject. Run client.close and create a new client before retrying.

4. Recover

Section titled “4. Recover”

There is no tooling yet to re-send a single message to its destination. Before the deadline, raise it with group::compliance in #sec-compliance-alerts, with the stream sequences and the deadline.

If the flag needs to be turned off to stop more messages from reaching the limit, disable audit_event_streaming_nats_consumer_redelivery. The next provisioning run, within 5 minutes, restores unlimited deliveries. Messages that already reached the limit stay stopped.