Skip to content

Customer data export and import#

A customer-data package (.bmpkg) holds customers, their stages (environments), the application versions those stages run, their tool schedules and, if you ask for them, the deploy, server and exception logs. It is how customer configuration travels between CommandCenter environments: export here, carry the file, import there. The bmo CLI reads and writes the same packages (bmo manage customer-data …).

Doing it: the user guide walks through export, checking a package and import step by step.

Since ADR 0005 Phase 4 the export runs in CommandCenter (WebApi) as the background task Export customer data, and the package lands in shared blob storage — so the download works from any replica, and the task page shows the steps as they happen.

What you can export#

Selection What ends up in the package
A selection of customers Those customers, every stage of each, the versions and schedules they refer to
One stage That stage and its customer only
Nothing selected Every customer

Log tables are opt-in — Deploy, Server and Exception — and can be limited to the last so many days. Server logs bring the servers and server versions they refer to along, so the importer can link them again. Configuration is always read as one consistent snapshot; a stage that changes while the export runs is exported as it was when the export started.

Context#

flowchart LR
    operator(Operator)

    subgraph azure [Azure]
        direction TB
        bff((BFF<br/>YARP gateway))
        webapi((CommandCenter<br/>WebApi · 1-10 replicas))
        db[|borders:tb|CommandCenter db<br/>TblCustomer · TblCustomerStage · logs<br/>TblBackgroundTask · TblCustomerDataExport]
        blob[(Blob storage<br/>customer-data container<br/>exports/{taskId}.bmpkg)]

        bff -- "/api/bmo/customers/{env}/export/** · import/**" --> webapi
        webapi -- "snapshot read" --> db
        webapi -- "upload · ranged download · delete" --> blob
    end

    cli[bmo CLI]

    operator -- "browser: start, task page, download" --> bff
    cli -. "Phase 5: same routes" .-> bff

    classDef boundary fill:none
    azure:::boundary
Hold "Alt" / "Option" to enable pan & zoom

Data flow#

sequenceDiagram
    autonumber
    actor Op as Operator
    participant SPA as Customers page / task page
    participant API as WebApi (any replica)
    participant Q as TblBackgroundTask
    participant W as Task worker (the replica that claims it)
    participant DB as CommandCenter db
    participant Blob as customer-data container

    Op->>SPA: Export (customers or a stage, log options)
    SPA->>API: POST /api/bmo/customers/{env}/export?customers=…&logs=…&logDays=…
    API->>DB: the names and the stage exist?
    API->>Q: queue customer-data-export (subject: the names)
    API-->>SPA: 202 {taskId, statusUrl, downloadUrl}
    SPA->>Op: task page /background-tasks/{taskId}
    W->>Q: claim
    W->>DB: discover: keys, one snapshot transaction
    W->>W: export-configuration, export-*-log → temp .bmpkg
    W->>Blob: upload-package (SHA-256 computed first)
    W->>DB: record: TblCustomerDataExport (bytes, digest, counts)
    W->>Q: Succeeded
    loop until the package is ready
        SPA->>API: GET /api/background-tasks/{taskId}
    end
    Op->>SPA: Download package
    SPA->>API: GET …/export/{taskId}/download (Range welcome)
    API->>Blob: ranged read
    API-->>SPA: 200/206, X-Content-Sha256
    SPA->>SPA: hash the bytes, compare, then save
Hold "Alt" / "Option" to enable pan & zoom

Once the task has succeeded, its page shows the package — size, digest, what it holds — with Download package and Remove package. The browser checks the downloaded bytes against the digest the server sends before saving the file, so a truncated download is reported rather than kept.

How long a package is kept#

A package is kept for two hours after the export finishes, or after it was last downloaded, whichever is later. The customer-data-prune scheduled job (every 15 minutes, see Config → Scheduled jobs) removes older packages, and the task page then says the package has been removed. The task and its log stay for the usual 30 days. Start a new export to download the data again.

The retention is a setting: CustomerData:Export:Retention, a time span.

User journey#

journey
    title Moving a customer to another environment
    section Here
      Tick the customer, Export: 5: Operator
      Watch the steps: 4: Operator
      Download, digest checked: 5: Operator
    section There
      Import the package: 4: Operator
      Review the analysis: 4: Operator
      Commit, watch it in the dialog: 4: Operator
Hold "Alt" / "Option" to enable pan & zoom

Importing a package#

Import runs in three stages: upload the file, look at it, commit it. Only the commit changes anything, and it runs as the background task Import customer data.

  1. Upload. The file goes up in chunks of 4 MiB into shared storage. A chunk that fails is simply sent again; a chunk sent twice is not stored twice. An upload nobody finishes is removed after 24 hours (CustomerData:Import:SessionRetention).
  2. Look before you import. Two read-only views, neither touches the database:
    • Package details — what bmo manage customer-data show -v prints: where and when it was exported, its size and SHA-256, rows per table with the oldest and newest log entry, the customers and stages it holds, and the tool schedules that will start running once imported.
    • Analysis — what would happen here: which versions, customers and stages already exist.
  3. Commit, choosing what happens to customers and stages that already exist:
On conflict Existing customer Existing stage Its schedules and logs
Skip kept as it is kept as it is not imported — they are already here
Fail the import is refused before anything is written
Replace kept, with its other stages removed and imported again from the package the package's are imported

Replace only touches the stages in the package — never the customer's other stages — and refuses a stage that is active or online or still being cloned, the same rule as deleting one; take it offline and deactivate it first. What it does to a replaced stage's related rows:

  • server log rows and processing status rows are deleted;
  • deploy logs, exception logs and tool-schedule run history are kept but unlinked, as when a stage is deleted;
  • the stage's tool schedules are deleted, because the package brings the stage's schedules with it.

instanceOverride and databaseOverride apply to the stages the import inserts, replaced stages included, since a replaced stage is inserted again; this is how a package exported from ACC lands on PRD's SQL instance. The Customers-page dialog sends only instanceOverride, fixed to the Settings page's default SQL Server instance (the legacy GetDefaultInstance, which also fills in a stage added by hand), and never databaseOverride; both free choices are for the bmo command-line tool. Without that setting the dialog refuses an import that would insert a stage. The route's {environment} is the Settings page's Stage. Imported active schedules get their next run moment recalculated, so they do not all fire the moment they arrive.

Which packages it accepts#

In short: version-2 packages only — see package versions below for what that means.

A package carries a schema version in its manifest; this CommandCenter reads version 2, which everything since BMO's customer-data export writes. Anything else — or a file with no manifest — is refused when you look at it, when you commit it, and by the task itself, with the advice to export it again.

There is no check on which CommandCenter produced the package. Tables are matched column by column instead: a column the package has and this database does not is ignored, a column the package lacks gets the database default. Only a required column without a default, missing from the package, makes the import fail.

sequenceDiagram
    autonumber
    actor Op as Operator
    participant API as WebApi (any replica)
    participant Row as TblCustomerDataUpload
    participant Blob as customer-data container
    participant W as Import task (claiming replica)
    participant DB as CommandCenter db

    Op->>API: POST …/import/sessions
    API->>Row: new session
    loop each 4 MiB chunk
        Op->>API: PUT …/chunks (Content-Range)
        API->>Blob: stage block at offset
        API->>Row: advance, only if ReceivedBytes = offset
    end
    API->>Blob: commit block list (last chunk)
    Op->>API: GET …/package, POST …/analyze (read-only)
    Op->>API: POST …/commit?onConflict=…
    API->>DB: check conflicts
    API->>Row: Open → Committing (once)
    API-->>Op: 202 {taskId}
    Note over Op,API: the Import dialog now follows the task (GET /api/background-tasks/{taskId})
    W->>Blob: download package
    W->>DB: configuration, one transaction
    W->>Row: key maps, same transaction
    loop each log batch
        W->>DB: SqlBulkCopy batch
        W->>Row: line counter, same transaction
    end
    W->>Row: Completed
    Op->>API: GET …/result
Hold "Alt" / "Option" to enable pan & zoom

The import task#

  1. Download the package and check its version.
  2. Compare with what exists — and refuse as the conflict choice says.
  3. Clear the stages being replaced — their server log rows, in batches (Replace only).
  4. Import customers, stages, versions and schedules, in one transaction.
  5. Import the deploy log, the server log, the exception log, in batches.
  6. Complete the import: the step's message and the session carry the counts.

A replica that stops mid-import has another pick it up. It does not start over: step 4 is recorded together with its own data and never runs twice, and each log batch is recorded with the batch, so the next attempt carries on after the last one committed. A step that fails ends the task; what was committed before it stays, and the task page and the session say where it stopped.

Package versions#

Every package says which format it is in: SchemaVersion in its /manifest.json. There have been two. Both are a .bmpkg file — an OPC (zip) container with the same manifest fields (SchemaVersion, SourceEnvironment, ExportedAt, CustomerFilter, TablesIncluded) and the same four core tables. They differ in how the tables are stored and in what a package can hold.

Version 1 Version 2
Written by BMO releases before bmo-v1.7.0 (May – June 2026) BMO bmo-v1.7.0 (19 June 2026) and later; CommandCenter since ADR 0005 Phase 4d
Table parts /{Table}.json — one indented JSON array per table /{Table}.ndjson.gz — gzip, one JSON object per line
How it was written whole tables loaded into memory, in one request streamed row by row from one database snapshot, as a background job
Log tables all or nothing (--include-logs): deploy log, server log, with their servers and server versions any of deploy, server and exception log, each on its own
Exception log never included optional
Log history limit none — all history optional, the last N days
Readable by CommandCenter no yes

Version 2 replaced version 1 because a large customer's logs no longer fitted in one request's memory; the exception log and the day limit came with it.

Telling them apart#

  • In CommandCenter, an upload of a version-1 package is refused at its first look — package details, analysis, commit — with "Package schema version 1 is not supported. Re-export it with the current CommandCenter." Nothing is imported.
  • With the CLI, bmo manage customer-data show <file> prints the schema version on its first lines.
  • By hand, open a copy of the file as a zip: version 1 has TblCustomer.json, version 2 has TblCustomer.ndjson.gz.

A version-1 file you still need#

There is no converter. Export the data again from its source environment with a current exporter — CommandCenter, or bmo v1.7.0 or later — and import the new package. If the source environment no longer exists, only a BMO release before bmo-v1.7.0 can read the file.

Same version, different exporter#

CommandCenter and BMO write version 2 byte for byte the same way, and each reads the other's packages. What can differ is the content:

  • CustomerFilter holds one customer name in a BMO package. A CommandCenter export of a selection holds all the names, comma-separated (Acme, Beta); an export of every customer leaves it out. It is informational — shown by show, never read by the import.
  • A stage export (CommandCenter only) holds one customer with just that stage, its schedules and its logs.

Same package, different importer#

The format is the same, but the two importers do not treat existing data the same way. Import CommandCenter packages with CommandCenter:

On conflict BMO's importer CommandCenter's importer
Skip keeps existing stages, but imports their schedules and logs again — duplicates, and it can stop half-way on the exception log keeps existing stages and imports nothing under them
Replace deletes the existing customer with all its stages, including stages that are not in the package keeps the customer and its other stages; replaces only the packaged stages, and only if they are inactive and offline
Fail stops at the first conflict, after writing what came before it refuses before writing anything
A failure half-way the upload cannot be committed again; upload it again what was committed stays; a replica that stops is resumed, not restarted

If something fails#

The task page shows the failing step. Causes worth knowing:

  • Snapshot isolation is off on the CommandCenter database. The export's discover step, or the import's configuration step, fails with the ALTER DATABASE … SET ALLOW_SNAPSHOT_ISOLATION ON statement to run. Both BMO and CommandCenter need it.
  • No customer-data storage on this host (CustomerData:Storage not configured). Export and upload are refused up front with 503; nothing is queued.
  • A stage became active or online between commit and import, under Replace: the task refuses it and writes nothing.

A replica that stops mid-export has the task picked up by another, which starts again from the first step; the export reads only, so that is safe, and the package is simply produced again.