Pulp Repository Metadata Signing Keys

Pulp supports two different types of GPG signatures: packages and repository metadata.

This document covers Pulp-specific implementation details for managing repository metadata signing keys.

For the generic process, see manage repository metadata signing keys.

For package signing, see manage package signing keys.

Overview

Pulp signs repository metadata using a private key that is generated externally and configured in the app. This is a security feature that gives our users certainty that the repository metadata was generated by us.

We manage the key externally by providing the private key to Pulp using a Kubernetes secret. This secret is synced from Vault, which is ultimately where the key lives and where any changes need to take place.

Key Location

The private key lives in Vault under the path k8s/<env>-gitlab-gke/pulp/gpg:

• Production environment: k8s/ops-gitlab-gke/pulp/gpg
• Test environment: k8s/pre-gitlab-gke/pulp/gpg

Pulp-Specific Process

This process should be carried out by a member of the Build team.

See manage repository metadata signing keys for the generic process steps. The following are Pulp-specific implementation details:

1. Request Access

• System(s): Okta Group Membership
• System Name: Okta Group: Team - Distribution - Pulp Repository Metadata Signing Key
• Justification: Temporary group membership required to update the Pulp repository metadata signing key in Vault.

2. Generate or Extend GPG Key

If extending an existing key, you can obtain the current private key from https://vault.gitlab.net by navigating to k8s/<env>-gitlab-gke/pulp/gpg. If you see an access denied message, reach out to #it_help to confirm you were added to the correct group.

Pulp Deployment Steps

After completing the generic steps 1-2, follow these Pulp-specific deployment steps:

Caution

Before starting, create a Criticality 3 change request and get it approved. This process modifies signing services and republishes all Pulp repositories, which affects package installations for all GitLab users.

3. Update Key in System

Update Vault Secret

1. Open https://vault.gitlab.net and sign-in using Okta.
2. Navigate to the path: k8s/<env>-gitlab-gke/pulp/gpg.
3. Click on Create new version.
4. Update the value of private_key with the contents of your exported private key.
5. Click Save.
6. Take note of the version number (displayed next to Create new version). You’ll need this for the next step.

Update ArgoCD Applications

Update the ArgoCD Applications repository with the new secret version:

1. Update the version number to the new Vault secret version from the previous step in services/pulp/env/<env>/values-vault-secrets.yaml
2. File an MR with the above changes and have someone in #infrastructure_platforms review/approve/merge it.

Publish the public key

1. Export the public key
2. Upload the public key to the pulp-resources-automation repository:
   • Go to files/<env>/gpgkey (note that, the ops environment is the production instance)
   • Upload the public key with the name gpg.key

Create new signing services and republish all repositories

Pulp manages the repository metadata signing key using signing services:

• rpm-signing-service
• apt-signing-service

They are designed to be immutable. Thus, when the key expires, we cannot update these signing services, but creating new ones. The tricky thing is that, Pulp Operator has these signing key names hardcoded (we call them “canonical signing services”), thus we have to do the following:

• Create temporary signing services
• Use the temporary signing services for all repositories
• Delete and recreate the canonical signing services
• Use the recreated canonical signing services for all repositories

Obtain the fingerprint of the new key:

gpg --show-keys --with-colons /path/to/new-key.gpg | grep '^fpr' | cut -d: -f10

Set these variables for use throughout the process:

PULP_BASE_URL="https://pulp.gitlab.com"
PULP_USER="admin"
PULP_PASS="secret"
NEW_KEY_FINGERPRINT=$(gpg --show-keys --with-colons /path/to/new-key.gpg | grep '^fpr' | cut -d: -f10)
DATE_PREFIX=$(date +%Y-%m-%d)   # e.g. 2026-03-25

The script is available at scripts/pulp/pulp-repo-signing.sh.

3.1 — Create temporary signing services

All pulpcore-manager commands must be run inside a pulp-api pod:

kubectl exec -it <pulp-api-pod> -- bash

pulpcore-manager add-signing-service "${DATE_PREFIX}-apt-signing-service" \
  /var/lib/pulp/scripts/apt_script.sh \
  "$NEW_KEY_FINGERPRINT" \
  --class deb:AptReleaseSigningService

pulpcore-manager add-signing-service "${DATE_PREFIX}-rpm-signing-service" \
  /var/lib/pulp/scripts/rpm_script.sh \
  "$NEW_KEY_FINGERPRINT"

Verify both temporary signing services were created:

pulp --profile ops signing-service list \
  | jq '.[] | select(.name | startswith(env.DATE_PREFIX)) | {name, pubkey_fingerprint}'

3.2 — Prepare the repo list

Fetch all RPM and DEB repositories from Pulp and write one YAML file per repo into the repos/ directory:

./scripts/pulp/pulp-repo-signing.sh \
  -m prepare \
  -b "$PULP_BASE_URL" \
  -u "$PULP_USER" \
  -p "$PULP_PASS" \
  -o repos/

3.3 — Update all repos with the temporary signing services

Assign the temporary signing services to all repositories:

./scripts/pulp/pulp-repo-signing.sh \
  -m update \
  -b "$PULP_BASE_URL" \
  -u "$PULP_USER" \
  -p "$PULP_PASS" \
  -r "${DATE_PREFIX}-rpm-signing-service" \
  -d "${DATE_PREFIX}-apt-signing-service" \
  -o repos/

If any repos fail, check their log files:

find repos/ -name "*.log"

Re-run the same command to retry failed repos — already-updated repos are skipped automatically.

3.4 — Publish all repos with the temporary signing services

./scripts/pulp/pulp-repo-signing.sh \
  -m publish \
  -b "$PULP_BASE_URL" \
  -u "$PULP_USER" \
  -p "$PULP_PASS" \
  -d "${DATE_PREFIX}-apt-signing-service" \
  -o repos/

Re-run to retry any failures. Already-published repos are skipped automatically.

3.5 — Delete old APT publications

With DEB repositories, they signing key is assigned to each publication. Thus, we need to find and delete all DEB publications that still reference the old canonical apt-signing-service.

Note

Deleting publications does NOT delete any packages. It only removes the snapshot metadata. Package files are unaffected.

First get its href:

OLD_APT_SIGNING_HREF=$(pulp --profile ops signing-service list --name "apt-signing-service" \
  | jq -r '.[0].pulp_href // empty')

Then delete in parallel, paging through oldest first:

SIGNING_SERVICE="$OLD_APT_SIGNING_HREF"
PAGE_SIZE=100
offset=0

while true; do
  page=$(pulp --profile ops deb publication list --limit "$PAGE_SIZE" --offset "$offset" --ordering pulp_created 2>/dev/null \
    | grep -m1 '^\[')

  hrefs=$(echo "$page" | jq -r --arg svc "$SIGNING_SERVICE" '.[] | select(.signing_service == $svc) | .pulp_href')
  page_count=$(echo "$page" | jq 'length')

  if [[ -n "$hrefs" ]]; then
    echo "$hrefs" \
      | xargs -P 10 -I{} bash -c '
          if pulp --profile ops deb publication destroy --href "$1" >/dev/null 2>&1; then
            echo "Deleted: $1"
          else
            echo "FAILED: $1"
          fi' _ {}
  fi

  offset=$(( offset + PAGE_SIZE ))
  echo "Processed offset $offset"
  [[ "$page_count" -lt "$PAGE_SIZE" ]] && break
done

3.6 — Delete the old canonical signing services

Run inside a pulp-api pod (can reuse the same session from 3.1):

pulpcore-manager remove-signing-service apt-signing-service --class deb:AptReleaseSigningService
pulpcore-manager remove-signing-service rpm-signing-service

If either command fails with “still in use”, there are remaining publications referencing the old service. Re-run the publication deletion from 3.5 to find and delete them.

3.7 — Recreate the canonical signing services with the new key

pulpcore-manager add-signing-service apt-signing-service \
  /var/lib/pulp/scripts/apt_script.sh \
  "$NEW_KEY_FINGERPRINT" \
  --class deb:AptReleaseSigningService

pulpcore-manager add-signing-service rpm-signing-service \
  /var/lib/pulp/scripts/rpm_script.sh \
  "$NEW_KEY_FINGERPRINT"

The temporary signing services can be left in place as a historical record.

3.8 — Reassign all repos to the recreated canonical signing services

Re-run prepare to regenerate all repo files fresh:

./scripts/pulp/pulp-repo-signing.sh \
  -m prepare \
  -b "$PULP_BASE_URL" \
  -u "$PULP_USER" \
  -p "$PULP_PASS" \
  -o repos/

Update all repos to use the canonical signing services:

./scripts/pulp/pulp-repo-signing.sh \
  -m update \
  -b "$PULP_BASE_URL" \
  -u "$PULP_USER" \
  -p "$PULP_PASS" \
  -r "rpm-signing-service" \
  -d "apt-signing-service" \
  -o repos/

Publish all repos:

./scripts/pulp/pulp-repo-signing.sh \
  -m publish \
  -b "$PULP_BASE_URL" \
  -u "$PULP_USER" \
  -p "$PULP_PASS" \
  -d "apt-signing-service" \
  -o repos/

4. Validate Deployment

Check signing services have the correct fingerprints

pulp --profile ops signing-service list \
  | jq '.[] | select(.name == "apt-signing-service" or .name == "rpm-signing-service") | {name, pubkey_fingerprint}'

Check the fingerprint and expiry of the published keys

Verify the RPM repo key fingerprint and expiry date:

curl -sL https://pulp.gitlab.com/gitlab/gitlab-ee/el/10/x86_64/repodata/repomd.xml.key \
  | gpg --show-keys --with-colons - 2>/dev/null \
  | awk -F: '
      /^pub/ { expiry=$7 }
      /^fpr/ && expiry { cmd="date -r "expiry" +%Y-%m-%d"; cmd | getline d; close(cmd);
                         print "fingerprint: "$10"\nexpires:     "d; expiry="" }
    '

Verify the DEB repo signature fingerprint matches $NEW_KEY_FINGERPRINT:

curl -sL https://pulp.gitlab.com/gitlab/gitlab-ee/ubuntu/noble/dists/noble/InRelease \
  | gpg --verify - 2>&1 | grep -E 'fingerprint|using'

Both fingerprints should match $NEW_KEY_FINGERPRINT.

Set up GitLab repository

It is known that Debian Trixie is strict, so we need to test setting up the GitLab repository on a Debian Trixie machine.

Start a Docker container running Debian Trixie

docker run -it --rm --name debian-trixie debian:trixie /bin/bash

Then following our official installation documentation:

curl -s https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | bash
apt update

If the above command succeeds, then the new repository signing key matches the InRelease file.

Complete the Generic Process

After validation, continue with the remaining steps in the generic runbook:

• Step 5: Clean Up Local Copies
• Step 6: Revoke Access