Gemnasium is down
NOTE: This service is being turned down.
Case 1: the HTTP service doesn’t respond (not error 500)
Section titled “Case 1: the HTTP service doesn’t respond (not error 500)”Symptoms
Section titled “Symptoms”- The Dependency Scanning jobs fail because they can’t contact the Gemnasium API.
- https://deps.sec.gitlab.com/ doesn’t respond or fails with an error.
Pre-checks
Section titled “Pre-checks”Try connecting to https://deps.sec.gitlab.com/ , a page listing packages by technologies should display. If it doesn’t then the service is down.
Resolution
Section titled “Resolution”- Navigate to the GCP page for the Gemnasium web workload
- Its status shouldn’t be “running”.
If the status is not “running”, the “Events” tab might list the cause of the failure.
In any case, restart the workload. To achieve this, you can force a rolling update:
- Navigate to the Kubernetes production cluster.
- Click “connect” and then “run in cloud shell”.
- Run the command it auto fills in.
- Relabel the web workload to trigger a rolling update and restart the pods:
kubectl patch statefulset web -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"- The pods should restart immediately and the service should respond again on its HTTP(s) port in less than a minute.
Post-checks
Section titled “Post-checks”Browse https://deps.sec.gitlab.com/explore , the website should appear.
Case 2 : the HTTP service returns a 5xx error (likely a database error)
Section titled “Case 2 : the HTTP service returns a 5xx error (likely a database error)”Symptoms
Section titled “Symptoms”- The Dependency Scanning jobs fail because the Gemnasium service fails (error 500).
- https://deps.sec.gitlab.com/explore/packages/gem/browse fails to list any packages.
Pre-checks
Section titled “Pre-checks”Try connecting to https://deps.sec.gitlab.com/explore/packages/gem/browse , a page listing Gem packages by technologies should display. If it doesn’t then the service fails, most likely it can’t contact its database server.
Check the application logs to be sure:
- Navigate to the GCP page for the Gemnasium web workload
- Navigate to one of its pods.
- Navigate to the “Logs” tab, the application logs should reflect a problem with connecting to the database.
Check the database:
- Navigate to the GCP page for the Gemnasium database service
Resolution
Section titled “Resolution”If the database is stopped, fix the issue and restart it.
Restart the “web” workload if the service still doesn’t work:
- Navigate to the Kubernetes production cluster.
- Click “connect” and then “run in cloud shell”.
- Run the command it auto fills in.
- Relabel the web workload to trigger a rolling update and restart the pods:
kubectl patch statefulset web -p "{\"spec\":{\"template\":{\"metadata\":{\"labels\":{\"date\":\"`date +'%s'`\"}}}}}"Post-checks
Section titled “Post-checks”- Browse https://deps.sec.gitlab.com/explore/packages/gem/browse , Gem packages should be listed.
- Query the API:
curl -d '[{"type": "gem", "name": "nokogiri"},{"type": "gem", "name": "actionpack"}]' https://deps.sec.gitlab.com/api/advisories/qSeveral screens of JSON data should be returned.