Pulp Runbook - Functional Operations

Overview

GitLab uses Pulp to store and distribute RPM and DEB packages for different RHEL-based and Debian-based OS distributions. This runbook covers the functional operation perspective of Pulp and our usage patterns.

[[TOC]]

Core Concepts

Pulp uses three key components for package management:

Repository - A collection of packages that maintains multiple versions based on changes made during its lifetime.
Publication - A distributable snapshot of a repository pointing to a specific repository version. All our repositories are configured to automatically create a new publication whenever a new package is uploaded.
Distribution - The endpoint through which a repository is made accessible to users. Each distribution has a base path defined relative to the Pulp instance root. Distributions can point to either:
- A specific publication, or
- A repository (always pointing to the publication of the latest repository version) - this is what we use

Terminology note: To avoid confusion, this document uses “Pulp repository” and “Pulp distribution” when referring to Pulp-specific concepts. “OS distribution” refers to systems like “Debian Trixie”, “Ubuntu Noble”, or “AlmaLinux 9”.

Repository Structure

APT and RPM repositories differ in their structure and naming pattern, as described below. Pulp distributions and their corresponding repositories have the same names, and the name is indicative of the final URL where they get served from.

APT Repositories

Background: Theoretically, an APT repository can hold packages for different architectures and OS distributions, provided there are no duplicate packages.

Our challenge: Packages for a GitLab version created by omnibus-gitlab for different OS distributions all share the same version string but have different content (and checksums). This makes them duplicate packages that would overwrite each other in a single repository. We have 10 years of packages versioned this way that must remain accessible to customers.

Our solution: One Pulp repository per OS distribution for each component:

Example repositories: gitlab-gitlab-ee-debian-trixie, gitlab-gitlab-ce-ubuntu-noble
Each repository holds packages for different architectures (APT repositories segregate metadata by architecture)

Access pattern:

https://packages.gitlab.com/<organization>/<namespace>/<os>/<distribution>

Example sources.list entry:

deb https://packages.gitlab.com/gitlab/gitlab-ee/debian/trixie trixie main

RPM Repositories

Structure: Each architecture of each OS version gets its own Pulp repository. This allows serving packages for each architecture at different endpoints, saving users from downloading metadata for other architectures. This matches the standard structure used by upstream EL distributions.

Access pattern:

https://packages.gitlab.com/<organization>/<namespace>/<os>/<version>/<architecture>

Example repo configuration:

baseurl=https://packages.gitlab.com/gitlab/gitlab-ee/el/9/$basearch

Repository Management

Creating and Updating Repositories

Pulp repositories and distributions are managed by code.

To add a new OS distribution:

Edit the YAML configuration file
Add the new OS to the relevant components

Important: Repository deletion must be done manually to prevent accidental data loss.

Uploading Packages

Prerequisites:

Pulp CLI configured and authenticated
Credentials for a user account with upload permissions.

DEB Package Upload

pulp deb content upload \
  --file=<path_to_file> \
  --repository=<pulp-repository> \
  --distribution=<os-distribution> \
  --component=main

Example:

pulp deb content upload \
  --file=gitlab-ee_18.0.0-ee.0_amd64.deb \
  --repository=gitlab-gitlab-ee-debian-trixie \
  --distribution=trixie \
  --component=main

Note: The --distribution flag is required even though there’s a Pulp repository per OS distribution. This ensures Pulp properly generates metadata files for each OS distribution.

RPM Package Upload

pulp rpm content upload \
  --file=<path_to_file> \
  --repository=<pulp-repository>

Example:

pulp rpm content upload \
  --file=gitlab-ee-18.0.0-ee.0.el9.x86_64.rpm \
  --repository=gitlab-gitlab-ee-el-9-x86_64

Important: The uploaded file must match the architecture corresponding to the repository.

Note: RPM uploads don’t require --distribution or --component flags because RPM internals create metadata files from all packages in a repository without a concept of “distributions”.

User Management

Check user management documentation for details on creation and management of users and roles in Pulp.

Operational Commands Reference

List Repositories

pulp deb repository list --limit=1000
pulp rpm repository list --limit=1000

Note: Use --offset flag to implement pagination.

List Distributions

pulp deb distribution list --limit=1000
pulp rpm distribution list --limit=1000

Note: Use --offset flag to implement pagination.

List All Packages

pulp deb content list --limit=1000
pulp rpm content list --limit=1000

Note: Use --offset flag to implement pagination.

List All Versions of a Specific Package

pulp deb content list --package=gitlab-ee --limit=1000
pulp rpm content list --name=gitlab-ee --limit=1000

Note: Additional filters are available - check --help for each command.

Validate a Specific Package Version Exists

To verify a specific package version exists in Pulp’s content store:

Prerequisites:

Pulp CLI configured and authenticated

DEB Package Validation

pulp deb content list \
  --package=<package-name> \
  --version="<exact-version>" \
  --field version \
  --field relative_path

Examples:

Validate a stable release:

pulp deb content list \
  --package=gitlab-ee \
  --version="18.10.0-ee.0" \
  --field version \
  --field relative_path

Validate a release candidate:

pulp deb content list \
  --package=gitlab-ee \
  --version="18.10.0-rc42.ee.0" \
  --field version \
  --field relative_path

Validate a nightly build:

pulp deb content list \
  --package=gitlab-ee \
  --version="18.10+stable.2398694953.4845434f-0" \
  --field version \
  --field relative_path

Validate an auto-deploy package:

pulp deb content list \
  --package=gitlab-ee \
  --version="18.10.202603181806-6ee48bcc3b8.f3f3da516cc" \
  --field version \
  --field relative_path

RPM Package Validation

For RPM packages, both --version and --release fields are needed for precise matching:

pulp rpm content list \
  --name=<package-name> \
  --version="<version>" \
  --release="<release>" \
  --field version \
  --field release \
  --field location_href

Examples:

Validate a stable release:

pulp rpm content list \
  --name=gitlab-ee \
  --version="18.10.0" \
  --release="ee.0.el9" \
  --field version \
  --field release \
  --field location_href

Interpreting Results

Output	Meaning
`[]` (empty)	Package not in Pulp — never uploaded, or incorrect version string
Records present	Content exists in the store
Content exists but 404 on download	Package not added to repository/publication — see Troubleshooting

Version String Formats

GitLab DEB packages use these version formats:

Release Type	Example Version
Stable release	`18.10.0-ee.0`
Release candidate	`18.10.0-rc42.ee.0`
Stable branch auto-deploy	`18.10+stable.<pipeline_id>.<commit>-0`
Nightly build	`18.10.YYYYMMDDHHMM-<gitlab_commit>.<omnibus_commit>`
Release nightly	`18.10.0+rnightly.<pipeline_id>.<commit>-0`

GitLab RPM packages use separate version and release fields:

Release Type	Version	Release
Stable release	`18.10.0`	`ee.0.<os>` (e.g., `ee.0.el9`)
Release candidate	`18.10.0`	`rc42.ee.0.<os>` (e.g., `rc42.ee.0.el9`)

Note: The --version flag requires an exact match. If unsure of the exact version string, omit the --version flag and use --limit=20 to examine available versions.

Testing Package Downloadability

If you need to verify a package is actually downloadable (not just present in the content store):

For DEB packages:

# Get the relative_path from content list
pulp deb content list \
  --package=gitlab-ee \
  --version="18.10.0-ee.0" \
  --field relative_path

# Test download (replace with your distribution base URL and filename)
curl -I "https://packages.gitlab.com/gitlab/gitlab-ee/debian/pool/bookworm/main/g/gitlab-ee/<filename>.deb"

For RPM packages:

# Get the location_href from content list
pulp rpm content list \
  --name=gitlab-ee \
  --version="18.10.0" \
  --release="ee.0.el9" \
  --field location_href

# Test download (replace with your distribution base URL and filename)
curl -I "https://packages.gitlab.com/gitlab/gitlab-ee/el/9/x86_64/<filename>.rpm"

HTTP Status	Meaning
`200 OK`	Package is downloadable
`404 Not Found`	Package not in publication — see Troubleshooting

Note: All GitLab Pulp repositories have autopublish: true enabled. If a package exists in content but returns 404 on download, this typically indicates a publication or distribution configuration issue rather than a missing upload.

Create a Publication Manually

If auto-publish fails, manually create a publication:

pulp deb publication create --repository=<pulp-repository-name>
pulp rpm publication create --repository=<pulp-repository-name>

List Tasks by State

pulp task list --state=waiting
pulp task list --state=running
pulp task list --state=failed

Check Task Status

pulp task show --href=<pulp-task-href>