ClickHouse Cloud Service
- Alerts: https://alerts.gitlab.net/#/alerts?filter=%7Btype%3D%22clickhouse%22%2C%20tier%3D%22inf%22%7D
- Label: gitlab-com/gl-infra/production~“Service::Service::ClickHouseCloud”
Logging
Section titled “Logging”Troubleshooting Pointers
Section titled “Troubleshooting Pointers”- ErrorTracking main troubleshooting document
- ../product_analytics/clickhouse-backup-restore.md
- ../product_analytics/ssl-troubleshooting.md
- ClickHouse Cloud Failure Remediation, Backup & Restore Process
Summary
Section titled “Summary”ClickHouse Cloud is a managed verion of ClickHouse DB. It is managed by ClickHouse Inc.
We are adding ClickHouse Cloud databases to the GitLab.com staging and production environments.
Contact
Section titled “Contact”Any questions please reach out to the team in Slack via #f_clickhouse or tag @gitlab-org/maintainers/clickhouse
Provisioners
Section titled “Provisioners”For a list of team members responsible for provisioning and deprovisioning ClickHouse Cloud DBs see tech_stack.yml
ClickHouse Cloud Console
Section titled “ClickHouse Cloud Console”The management console for ClickHouse Cloud can be accessed at:
https://clickhouse.cloud/services
Log in via Google SSO.
You will need to have access granted by a ClickHouse Cloud admin to access the management console. An Access Request is a typical way of granting this permission. (example)
Network Interconnect
Section titled “Network Interconnect”Sidekiq and the Web tier will connect via HTTP on port 8443 via a public IP/host. Inbound connection can be allow listed via IPv4 ranges. See ClickHouse IP Access List Docs for details.
- GitLab Development Docs:
- ClickHouse HTTPS interface docs:
- ClickHouse Cloud is configured to accept HTTPS connections on port 8443.
GitLab.com Staging Database
Section titled “GitLab.com Staging Database”GitLab.com Staging Database has the following attributes:
Name: gitlab-com-staging Console URL: https://clickhouse.cloud/service/57fa3208-48a1-494b-9f44-ea895895a369
GitLab.com Production Database
Section titled “GitLab.com Production Database”GitLab.com Production Database has the following attributes:
Name: gitlab-com-production Console URL: https://clickhouse.cloud/service/ad02dd6a-1dde-4f8f-858d-37462fd06058
ClickHouse Cloud Backup & Restore
Section titled “ClickHouse Cloud Backup & Restore”Runbook for restoring a ClickHouse Cloud instance from a backup after failure of an instance. [Link]
Monitoring/Alerting
Section titled “Monitoring/Alerting”Built in Monitoring Dashboards can be accessed via the ClickHouse Cloud Console.
A dashboard for monitoring and alerting performance metrics has been configured in Runbooks, clickhouse-cloud-services
Alerting has been configured through Runbooks to alert into Slack channel #f_clickhouse_infra_alerts
Falling back to PostgreSQL
Section titled “Falling back to PostgreSQL”Several analytics features are implemented using ClickHouse at GitLab. Contribution Analytics was one of the first features to make use of ClickHouse; originally implemented using PostgreSQL, which performed poorly, the ClickHouse-based implementation provided a much better performance.
In case of an outage in ClickHouse cloud services, the impact would include:
- Slower load times for the Contribution Analytics pages.
- An outage in features that are implemented in ClickHouse only, for example: AI impact analytics and CI/CD analytics.
While the option is there to switch back to PostgreSQL by disabling ClickHouse, this could be helpful in restoring some features that are implmeneted in both PG and CH, but won’t be helpful for features that depend only on CH.
When we disable the analytics ClickHouse integration for features that exist in PG too (Contribution Analytics), we’re not creating a data split. Data is always present in PG and it’s the source of truth. When ClickHouse integration is disabled, the backend falls back to querying PG. Once the integration is switched back on, background workers will ensure data during the outage is synced to ClickHouse.