Skip to content

Background tasks#

Some operations take too long to answer a request with. CommandCenter runs those as background tasks: durable work items with a step-by-step log you can watch while they run.

Find them under Background tasks in the navigation. On the Customers page, the dialog in which you start one — an export, an import, deleting an environment — stays open and follows the task in place, with a link to its task page; closing that dialog does not stop the task.

What you see#

The list shows every task, newest first, filterable by status and type.

Status Meaning
Queued Accepted, not started. A task can be queued for a future date — see Scheduled for.
Running A replica has claimed it and is working through the steps.
Succeeded Every step completed.
Failed A step failed, or the replica running it stopped repeatedly. The reason is on the task page.
Cancelled Stopped before or between steps, either by you or because it could not be recovered.

Opening a task shows its steps as they happen — name, how long each took, and a running count for steps that process many rows. A step that is still going shows a spinner rather than being hidden, so a task that stops halfway still tells you where it got to.

The page updates live when it can. If your connection or a proxy will not stream, it falls back to refreshing every couple of seconds; the outcome is the same either way.

Running a scheduled task now#

A task that is Queued with a Scheduled for date in the future shows Run now on its page. It brings the task forward; a worker picks it up within a few seconds and it runs exactly as it would have on schedule. A task that is already due, running or finished has nothing to bring forward.

Cancelling#

Cancel task stops a task that has not finished.

  • A task that has not started yet stops immediately.
  • A running task stops between steps, not in the middle of one. That is deliberate: it leaves the data in a coherent state rather than half-finished. It may take a moment. The exception is a long database copy in a clone: it is interrupted, and the half-restored copy is removed.
  • A task that has already finished cannot be cancelled.

Deleting a customer environment#

This is the operation most people meet a task through. After you confirm, a dialog follows the deletion's steps on the Customers page; the environment itself has already left the list.

On a customer's environment panel, Delete environment appears only when the environment is both offline and inactive. To take a live environment out of use first, without deleting anything, call POST /api/customers/stages/{identifier}/deactivate, which sets both flags off; this is what bmo manage remove-stage --force did on BMO. You confirm by typing the environment's name, and the server checks again independently — if someone brought it back online while the task was queued, the deletion stops rather than proceeding.

What happens then, in order:

  1. The environment is looked up and re-checked.
  2. Its server log rows are deleted, in batches, with a running count.
  3. Deploy logs, tool schedules and exception logs are kept but unlinked from the environment; processing status rows are deleted.
  4. The environment itself is removed. It disappears from the Customers page.
  5. A second task is scheduled to drop the environment's database in seven days.

The seven-day window is your undo#

The environment's rows go immediately, so the deletion visibly takes effect. The database is kept for a week.

That second task appears in the list straight away as Queued, with Scheduled for showing the date it will run. Cancel it and the database is kept. Run now does the opposite: it ends the grace period and the database is dropped within seconds. Because that cannot be undone, you confirm it by typing the environment name. This replaces the old behaviour, where the only way to stop a deletion was to set the customer back online within seven days.

After it runs, the database is gone and only a restore from backup will bring it back.

What is deliberately left behind#

Two things are not cleaned up automatically, and are known:

  • Rows in CustomerStageAdmin, which lives in a different database.
  • Two caches — the analytics schema cache and the BenefitManager proxy's routing map — keep the deleted environment for a few hours until they expire on their own.

Cloning a customer environment#

Every environment on the Customers page has a Clone button — offline and inactive ones too, since clones are commonly made to investigate an environment that is down. The dialog suggests a name, URL and database name ({Name}-{n}, {Url}/{n}, {Database}_{n}, with the first free number), and two options, Set active and Set online, both off.

An existing database is never overwritten. While you type, the form checks the server and refuses to submit — and says why — when:

  • a database of that name already exists on the environment's SQL Server instance;
  • the customer already has an environment of that name, or any environment uses that URL, or any environment names that database;
  • a clone for any of those is already queued;
  • the CommandCenter login lacks the permissions below, or the instance cannot be reached.

What the task does, in order:

  1. Preflight: checks permissions on the instance.
  2. Verify target: the database must not exist. If a previous attempt of the same task left its own copy, that copy is kept (finished) or removed and redone (half-restored); anything else stops the task.
  3. Backup: a copy-only backup of the source database, striped over up to eight files. The source environment keeps running.
  4. Restore under the new name, on the same instance.
  5. Create stage: the environment appears on the Customers page, inactive and offline.
  6. Activate: switches on what you ticked (skipped if nothing was ticked).
  7. Cleanup: removes the backup files.

The new environment only appears once its database is complete, so it never points at a database the task did not create. While the task runs, the new environment cannot be deleted.

SQL Server permissions it needs#

On the source environment's instance, the CommandCenter login needs:

  • BACKUP DATABASE on the source database;
  • CREATE ANY DATABASE (the dbcreator role);
  • VIEW ANY DEFINITION.

Deleting the backup files uses xp_delete_file, which needs sysadmin. Without it the clone still succeeds; the Cleanup step lists the files left behind (*_{taskId}_Part{n}.bak in the instance's default backup directory, or BackgroundTasks:Clone:BackupDirectory), for someone to delete.

If a clone fails#

Nothing that already existed is touched. A failure before the restore leaves nothing behind but, possibly, backup files. A restore that fails or is cancelled has its half-restored copy removed. If the task fails after the restore — rare — the new database exists without an environment; the error names the instance and database, and its files carry the task id ({database}_{taskId}_…), so it can be recognised and dropped by hand.

A database drop that is still pending from an earlier deletion is refused if an environment uses that database name again by the time it runs.

Exporting customer data#

Export customer data produces a .bmpkg package of customers, their environments, the versions and tool schedules they refer to and, when asked, their deploy, server and exception logs. The customer data page has the full picture; the task itself runs these steps:

  1. Resolve customers and stages: takes one consistent snapshot of the database and works out which rows the selection covers.
  2. Export customers, stages, versions and schedules into the package.
  3. Export the deploy log, the server log, the exception log: one step each, skipped when that log was not requested; a running row count while it works.
  4. Upload the package to the customer-data storage, with its SHA-256 digest.
  5. Record the package: from here the Export dialog and the task page show the package with Download package.

The export only reads, so a replica that stops mid-way simply has another produce the package again. The package is kept for two hours after the export finishes or after it was last downloaded; the customer-data-prune job removes it after that and the task page says so.

Importing customer data#

Import customer data imports an uploaded package — see customer data for the conflict choices and what Replace touches. Its steps: Download the package, Compare with what exists, Clear the stages being replaced (Replace only), Import customers, stages, versions and schedules, one Import … log step per log table in the package, Complete the import. A retried task says which steps an earlier attempt already did, and a log step resumes after the last batch that attempt committed.

Retention#

Finished tasks are removed after 30 days by the background-task-prune scheduled job, which you can see and adjust on Config → Scheduled jobs. It ships in dry-run, so until an operator turns that off it reports what it would remove without removing anything.

Export packages have their own, much shorter retention (two hours, customer-data-prune, not in dry-run); the task that produced one stays for the 30 days, without the package. The same job removes import uploads 24 hours after their last activity, except one whose import is still running.

If something fails#

The task page shows the failing step and the error. Tasks are safe to be retried — a replica that stops unexpectedly has its work picked up by another, which starts again from the first step and skips what is already done. After three attempts a task is failed and left for a human.

A failed database drop is never retried automatically: the database is left in place and needs someone to look at it.