Skip to content
Runbooks

CI Mirrored Tables

The tables gitlab_main tables: namespaces and projects that reside on the main database, are partially copied to their corresponding gitlab_ci database tables ci_namespace_mirrors and ci_project_mirrors respectively. This even happens on single database mode.

Troubleshooting

Section titled “Troubleshooting”

When receiving alerts about inconsistent records errors, we need to investigate the type of the inconsistency. This can be checked from Kibana logs. For example, if it is inconsistent updating of the records, or we have a delayed deletion of the records.

Inconsistent Namespace Records

Section titled “Inconsistent Namespace Records”
Section titled “Links”

Investigating the types of inconsistency

Section titled “Investigating the types of inconsistency”
  • Check the logs attribute json.extra.database_ci_namespace_mirrors_consistency_check_worker.results.mismatches_details

  1. If the mismatches_details for the mismatched namespaces contains both source_table and target_table, then this means that both source and target records exist in the source table and target table, but syncing is not working as expected. Check the section Wrong update of the mirrored records in this page.

  2. If the mismatches_details for the mismatched namespaces contains only the source_table, but not the target_table, then this means that we are also not updating the records. Also check Wrong update of the mirrored records in this page.

  3. If the mismatches_details for the mismatched namespaces contains only the target_table, but not the source_table, then this means we are not deleting the target objects upon deletion of the source tables. Check Delayed deletion of the mirrored records in this page.

Inconsistent Project Records

Section titled “Inconsistent Project Records”
Section titled “Links”

Investigating the types of inconsistency

Section titled “Investigating the types of inconsistency”
  • Check the logs attribute json.extra.database_ci_project_mirrors_consistency_check_worker.results.mismatches_details

  1. If the mismatches_details for the mismatched projects contains both source_table and target_table, then this means that both source and target records exist in the source table and target table, but syncing is not working as expected. Check the section Wrong update of the mirrored records in this page.

  2. If the mismatches_details for the mismatched projects contains only the source_table, but not the update_table, then this means that we are also not updating the records. Check the section Wrong update of the mirrored records in this page.

  3. If the mismatches_details for the mismatched projects contains only the target_table, but not the source_table, then this means we are not deleting the target objects upon deletion of the source tables. Check Delayed deletion of the mirrored records in this page.

Delayed deletion of mirrored records

Section titled “Delayed deletion of mirrored records”

Wrong update of mirrored records

Section titled “Wrong update of mirrored records”
  • Check recent merged requests affecting these classes, and possibly revert them if necessary:
  1. app/models/namespaces/traversal/linear.rb
  2. app/models/namespace/traversal_hierarchy.rb
  3. app/models/namespace.rb
  4. app/models/project.rb
  5. app/models/namespaces/sync_event.rb
  6. app/models/projects/sync_event.rb

Team

Section titled “Team”

Reach out to the Tenant Scale in case of any extra help needed.

References

Section titled “References”