Data Insights Platform - Product Usage Data via Snowplow
Note, these environments are not receiving production traffic yet. We’re currently planning migrating our current Snowplow data streams over to these Data Insights Platform backed environments. See work epic for details.
Overview
Section titled “Overview”These Data Insights Platform instances help ingest product usage data (Snowplow) from GitLab instances across multiple environments, i.e. .com, Dedicated & self-managed and export it to S3/Snowflake - to serve our internal data & analytics teams.
Snowflake sits outside the scope of these environments. We only land data in
S3- which is then ingested into Snowflake async.
Following is a brief overview of its setup in these environments:
- production:
usagestats.gitlab.com - staging:
usagestats.staging.gitlab.com
Access to these clusters
Section titled “Access to these clusters”- All currently provisioned DIP instances should be accessible to on-call engineers. The access to these underlying GKE clusters follows our standard VPN-based tool chain as documented here.
- In case of other folks needing access to these environments, an access request needs to be created - for example.
Observability
Section titled “Observability”SLIs for Data Insights Platform within analytics environments is defined in a separate metrics catalog file and available on Grafana here.
Metrics
Section titled “Metrics”- production: Grafana dashboard
- staging: Grafana dashboard
Alerts
Section titled “Alerts”(work in progress)
Infrastructure & Networking
Section titled “Infrastructure & Networking”production: usagestats.gitlab.com
Section titled “production: usagestats.gitlab.com”| Entity | Details |
|---|---|
| Provider | GCP |
| GCP Project | analytics-eventsdot-prod |
| Region | us-east1 |
| Networks | eventsdot-prod-vpc / eventsdot-prod-subnet |
| DNS Names | usagestats.gitlab.com |
| Deployment configs | In config-mgmt repository In gitlab-helmfiles repository |
staging: usagestats.staging.gitlab.com
Section titled “staging: usagestats.staging.gitlab.com”| Entity | Details |
|---|---|
| Provider | GCP |
| GCP Project | gl-analy-evtsdot-stg-0d67dbbc |
| Region | us-east1 |
| Networks | events-dot-stg-vpc / events-dot-stg-subnet |
| GKE cluster | events-dot-stg |
| DNS Names | usagestats.staging.gitlab.com |
| Deployment configs | In config-mgmt repository In gitlab-helmfiles repository |