Pulp Runbook - Functional Operations

Overview

GitLab uses Pulp to store and distribute RPM and DEB packages for different RHEL-based and Debian-based OS distributions. This runbook covers the functional operation perspective of Pulp and our usage patterns.

Core Concepts

Pulp uses three key components for package management:

1. Repository - A collection of packages that maintains multiple versions based on changes made during its lifetime.

2. Publication - A distributable snapshot of a repository pointing to a specific repository version. All our repositories are configured to automatically create a new publication whenever a new package is uploaded.

3. Distribution - The endpoint through which a repository is made accessible to users. Each distribution has a base path defined relative to the Pulp instance root. Distributions can point to either:

   • A specific publication, or
   • A repository (always pointing to the publication of the latest repository version) - this is what we use

Terminology note: To avoid confusion, this document uses “Pulp repository” and “Pulp distribution” when referring to Pulp-specific concepts. “OS distribution” refers to systems like “Debian Trixie”, “Ubuntu Noble”, or “AlmaLinux 9”.

Repository Structure

APT and RPM repositories differ in their structure and naming pattern, as described below. Pulp distributions and their corresponding repositories have the same names, and the name is indicative of the final URL where they get served from.

APT Repositories

Background: Theoretically, an APT repository can hold packages for different architectures and OS distributions, provided there are no duplicate packages.

Our challenge: Packages for a GitLab version created by omnibus-gitlab for different OS distributions all share the same version string but have different content (and checksums). This makes them duplicate packages that would overwrite each other in a single repository. We have 10 years of packages versioned this way that must remain accessible to customers.

Our solution: One Pulp repository per OS distribution for each component:

• Example repositories: gitlab-gitlab-ee-debian-trixie, gitlab-gitlab-ce-ubuntu-noble
• Each repository holds packages for different architectures (APT repositories segregate metadata by architecture)

Access pattern:

https://packages.gitlab.com/<organization>/<namespace>/<os>/<distribution>

Example sources.list entry:

deb https://packages.gitlab.com/gitlab/gitlab-ee/debian/trixie trixie main

RPM Repositories

Structure: Each architecture of each OS version gets its own Pulp repository. This allows serving packages for each architecture at different endpoints, saving users from downloading metadata for other architectures. This matches the standard structure used by upstream EL distributions.

Access pattern:

https://packages.gitlab.com/<organization>/<namespace>/<os>/<version>/<architecture>

Example repo configuration:

baseurl=https://packages.gitlab.com/gitlab/gitlab-ee/el/9/$basearch

Repository Management

Creating and Updating Repositories

Pulp repositories and distributions are managed by code.

To add a new OS distribution:

1. Edit the YAML configuration file
2. Add the new OS to the relevant components

Important: Repository deletion must be done manually to prevent accidental data loss.

Uploading Packages

Prerequisites:

• Pulp CLI configured and authenticated
• Credentials for a user account with upload permissions.

DEB Package Upload

pulp deb content upload \
  --file=<path_to_file> \
  --repository=<pulp-repository> \
  --distribution=<os-distribution> \
  --component=main

Example:

pulp deb content upload \
  --file=gitlab-ee_18.0.0-ee.0_amd64.deb \
  --repository=gitlab-gitlab-ee-debian-trixie \
  --distribution=trixie \
  --component=main

Note: The --distribution flag is required even though there’s a Pulp repository per OS distribution. This ensures Pulp properly generates metadata files for each OS distribution.

RPM Package Upload

pulp rpm content upload \
  --file=<path_to_file> \
  --repository=<pulp-repository>

Example:

pulp rpm content upload \
  --file=gitlab-ee-18.0.0-ee.0.el9.x86_64.rpm \
  --repository=gitlab-gitlab-ee-el-9-x86_64

Important: The uploaded file must match the architecture corresponding to the repository.

Note: RPM uploads don’t require --distribution or --component flags because RPM internals create metadata files from all packages in a repository without a concept of “distributions”.

User Management

Check user management documentation for details on creation and management of users and roles in Pulp.

Operational Commands Reference

List Repositories

pulp deb repository list --limit=1000
pulp rpm repository list --limit=1000

Note: Use --offset flag to implement pagination.

List Distributions

pulp deb distribution list --limit=1000
pulp rpm distribution list --limit=1000

Note: Use --offset flag to implement pagination.

List All Packages

pulp deb content list --limit=1000
pulp rpm content list --limit=1000

Note: Use --offset flag to implement pagination.

List All Versions of a Specific Package

pulp deb content list --package=gitlab-ee --limit=1000
pulp rpm content list --name=gitlab-ee --limit=1000

Note: Additional filters are available - check --help for each command.

Create a Publication Manually

If auto-publish fails, manually create a publication:

pulp deb publication create --repository=<pulp-repository-name>
pulp rpm publication create --repository=<pulp-repository-name>

List Tasks by State

pulp task list --state=waiting
pulp task list --state=running
pulp task list --state=failed

Check Task Status

pulp task show --href=<pulp-task-href>