Canceling running imports
When to use this runbook: a user (or their project) is generating load through an import, typically during an abuse-driven incident, and you have blocked or are about to block the user.
Summary
Section titled “Summary”To stop an import, cancel it. Canceling marks the import as canceled. Jobs that have not started yet exit when they run, and no further stages are scheduled.
Blocking or banning a user cancels their in-flight imports automatically (gitlab-org/gitlab!262867). The block or ban enqueues Users::CancelImportsForBlockedUserWorker, which cancels:
- Project imports in
scheduledorstartedstate for projects created by the user (pull mirrors are skipped). - Direct Transfer migrations in
createdorstartedstate owned by the user.
Use the manual steps below when:
- You need to verify the automatic cancel happened. The worker is low urgency, so allow it a moment to run. Its result is logged in the Sidekiq logs as
json.extra.users_cancel_imports_for_blocked_user_worker.canceled_project_importsandjson.extra.users_cancel_imports_for_blocked_user_worker.canceled_bulk_imports. In Kibana, filter onjson.class: Users::CancelImportsForBlockedUserWorker. A count of 0 can be correct. - The import is not covered by the worker, for example the project was created by a different user than the one you blocked, or you are canceling an import without blocking anyone.
- The user was deactivated or LDAP-blocked rather than blocked or banned. Those do not trigger the worker.
What canceling does and does not do
Section titled “What canceling does and does not do”- Queued jobs stay in Sidekiq. They exit when they run, so queue depth only drops as they drain.
- GitHub, Bitbucket Cloud, Bitbucket Server and Direct Transfer imports stop shortly after. Their workers check the import state between units of work.
- Repository by URL, manifest, FogBugz, Gitea and file-based imports are not interrupted once running. Each runs inside a single
RepositoryImportWorkerthat does not check the import state after it starts, so a clone or fetch in progress keeps loading Gitaly until it finishes. If the load continues after canceling, find and kill the job, see the Sidekiq survival guide for SREs. - Canceling cannot be undone. The project’s
import_datais deleted and a canceled import cannot be rescheduled, so unblocking the user does not resume it, whether the import was canceled manually or by the block. The partially imported project stays. Deleting it is a separate decision.
Prerequisites
Section titled “Prerequisites”- Rails console access on GitLab.com, see granting Rails or DB access.
- The username of the blocked user, or the full path of the project (for example from Gitaly errors or Kibana
json.meta.project).
-
Open a Rails console.
-
List the in-flight imports.
Project imports (GitHub, Bitbucket, Gitea, FogBugz, file-based, manifest, repository by URL) and Direct Transfer migrations. Set
scopeandbulk_import_scopeby project or by user:# By project:project = Project.find_by_full_path('<namespace/project>')scope = ProjectImportState.where(project: project)bulk_import_scope = BulkImport.where(id: BulkImports::Entity.where(project_id: project.id).select(:bulk_import_id))# By user:user = User.find_by_username('<username>')scope = ProjectImportState.joins(:project).where(projects: { creator_id: user.id })bulk_import_scope = BulkImport.where(user_id: user.id)Then list them:
# Pull mirrors share ProjectImportState, so leave them out: canceling also deletes the project's import_data.states = scope.where(status: %w[scheduled started]).preload(:project).to_a.reject { |state| state.project.mirror? }states.each { |s| puts [s.project.full_path, s.project.import_type, s.status].join("\t") }; nilDirect Transfer migrations:
bulk_imports = bulk_import_scope.with_status(:created, :started).to_abulk_imports.each { |b| puts [b.id, b.status_name, b.entities.count].join("\t") }; nil -
Cancel them.
cancelreturnstruewhen the import was canceled:states.each { |s| puts "#{s.project.full_path}: #{s.cancel}" }; nilbulk_imports.each { |b| puts "#{b.id}: #{b.cancel}" }; nilCanceling a
BulkImportalso cancels all its entities.falsemeans the import was not canceled, for example because it finished after you listed it. Check its status withreload.status. -
Run the listing from step 2 again. It should come back empty. If the load continues, see what canceling does and does not do.