troubleshooting

Troubleshooting Pulp

The commands in this runbook use the Pulp CLI. To install and configure it, see Pulp CLI.

Uploaded package not available in repository

Ensure that a content has been created for the uploaded package

pulp deb content list --package=<package-name> --version=<package-version> --architecture=<architecture>
pulp rpm content list --name=<package-name> --version=<package-version> --arch=<architecture>

If not, upload the package again, and ensure the task has completed successfully.

Get href of the latest version of the repository

pulp deb repository show --name=<repository-name> | jq -r '.latest_version_href'
pulp rpm repository show --name=<repository-name> | jq -r '.latest_version_href'

Check if package is in the latest version of the repository

pulp deb content list --package=<package-name> --version=<package-version> --architecture=<architecture> --repository-version=<repository-version-href>
pulp rpm content list --name=<package-name> --version=<package-version> --arch=<architecture> --repository-version=<repository-version-href>

If not, add the content to the repository

pulp deb repository content add --repository=<repository_name> --package-href=<package-content-href>
pulp rpm repository content add --repository=<repository_name> --package-href=<package-content-href>

Check if a publication exists that points to the latest version of the repository

pulp deb publication list --repository-version=<latest-repository-version-href>
pulp rpm publication list --repository-version=<latest-repository-version-href>

If not, create a publication manually

pulp deb publication create --repository=<repository-name>
pulp rpm publication create --repository=<repository-name>

Verify that a publication exist with the content

pulp deb publication list --content='["<package-content-href>"]'
pulp rpm publication list --content='["<package-content-href>"]'

GPG verification fails with 404 for repomd.xml.asc

Package installation from an RPM repository fails with an error like this:

Errors during downloading metadata for repository 'gitlab_pre-release':
  - Status code: 404 for https://pulp.gitlab.com/gitlab/pre-release/el/8/aarch64/repodata/repomd.xml.asc
Error: Failed to download metadata for repo 'gitlab_pre-release': GPG verification is enabled, but GPG signature is not available.

A 404 for repomd.xml.asc while repomd.xml downloads correctly means the served publication is not signed. Pulp only generates repomd.xml.asc when it creates the publication with a metadata signing service.

Check if the repository has a metadata signing service:

pulp rpm repository show --name=<repository-name> | jq -r '.metadata_signing_service'

If the value is null, the repository is detached from the signing service. Every new publication for this repository is unsigned.

Check that the canonical signing service exists:

pulp signing-service list | jq -r '.[] | [.name, .pulp_created] | @tsv'

If rpm-signing-service is missing, recreate it first. See manage repository metadata signing keys. A recent pulp_created timestamp on a canonical signing service shows that someone recreated it. Recreation does not re-attach the service to repositories.

Re-attach the signing service to all repositories.

Run a pipeline on main in pulp-resources-automation on ops.gitlab.net. No CI variables are needed. The repositories:pre and repositories:ops jobs detect the missing signing service and update every repository.

Create new publications for the repositories that serve unsigned metadata.

Re-attaching the signing service does not republish. Autopublish only fires on new content, so an unsigned publication stays live until a new one replaces it.

For a single repository:

pulp rpm publication create --repository=<repository-name>

For a set of repositories, iterate over the repository names. Adjust the name prefix to select the affected repositories. Set --limit higher than the total repository count, or the list is incomplete. The --background flag dispatches the task without waiting, so Pulp workers process the publications in parallel:

pulp rpm repository list --limit 5000 \
  | jq -r '.[] | select(.name | startswith("gitlab-pre-release-"))
                | select(.name | endswith("SRPMS") | not) | .name' \
  | while IFS= read -r repo; do
      service=$(pulp rpm repository show --name="$repo" | jq -r '.metadata_signing_service // "null"')
      if [[ "$service" == "null" ]]; then
        echo "SKIP $repo: signing service not attached"
        continue
      fi
      echo "Publishing $repo"
      pulp --background rpm publication create --repository="$repo"
    done

Check that the dispatched tasks complete:

pulp task list --name pulp_rpm.app.tasks.publishing.publish --state-in waiting --state-in running --limit 5000 | jq length
pulp task list --name pulp_rpm.app.tasks.publishing.publish --state failed --limit 10 | jq -r '.[] | .error.description'

Do not create publications before the signing service is re-attached. A publication created from a detached repository is unsigned and extends the outage.

The loop above excludes SRPM repositories because the automation normally creates their publications. However, the automation only creates an SRPM publication when none exists for the latest repository version. If an SRPM repository received an unsigned publication while detached, a pipeline rerun does not replace it. Create a publication for it manually with pulp rpm publication create.

Verify that the signature is served.

The distribution shares its name with the repository. Get the distribution path from its base_path, for example gitlab/pre-release/el/8/aarch64:

pulp rpm distribution show --name=<repository-name> | jq -r '.base_path'

curl -sL -o /dev/null -w '%{http_code}' https://pulp.gitlab.com/<base-path>/repodata/repomd.xml.asc

Expect 200. Private repositories need authentication, for example -u <user>:<token>.