Benefits Orchestrator (`bmo`) — Setup and Operations#

bmo is the operator CLI for the Benefits deployment system. It drives one or more Agents — uploading packages, triggering SQL operations via the Command Center service, managing versions, and querying environment state. All configuration lives in appsettings.json alongside the executable, which you edit interactively via bmo config.

Prerequisites#

Windows (x64)
Network access to each Agent's HTTP endpoint (default port 5100)
HTTP access to the Command Center service (default port 8090) — see commandcenter-setup-and-operations.md
Azure Blob Storage access if using --via blob deploys

Installation#

Developer workstation#

Run Register-BMO.ps1 from the repo root. It publishes the project and sets a bmo alias in your current session:

.\Register-BMO.ps1

Operations box#

Copy the published output of Benefits.Orchestrator to a local folder (e.g. C:\Tools\bmo\). bmo.exe and its appsettings.json must sit in the same directory — settings are read from the executable's location.

Add the folder to PATH so bmo is accessible from any prompt.

First-run: `bmo config init`#

Before using any other command, run the interactive setup wizard:

bmo config init

The wizard covers seven steps:

Step	What you configure
1	`VersionsPath` — local folder where deployment zip files are kept (default `F:\Versions`)
2	Command Center service — `BaseUrl` (e.g. `https://cc.internal:8090`) and auth mode + credentials for outbound calls to Command Center
3	Agent client auth — `ApiKey`, `mTLS`, or `EntraId` + credentials for outbound calls to Agents
4	Package store — Azure Blob `AccountUri` and container name (skip if not using blob transport)
5	Service Bus — namespace and topics (skip if using HTTP transport)
6	Agents — name + `BaseUrl` for each target server (e.g. `http://vmbenefitsbmp1:5100`)
7	Environments — `RequiresAcceptedStatus` per environment (SQL connection strings no longer configured here — see note below)

Note on step 7: In v2.0.0 the SQL connection strings moved to the Command Center service's appsettings.json. bmo communicates with SQL exclusively via the Command Center HTTP API. The connection string prompt in bmo config init step 7 is retained for backward compatibility but the values are not used by bmo; configure ConnectionStrings.CommandCenter_<Env> in the Command Center service instead.

When finished, verify with:

bmo config show    # prints active config with secrets redacted

Managing config after first-run#

All subcommands below edit appsettings.json interactively:

bmo config agent add|remove|list         # manage Agent endpoints
bmo config environment add|remove|list   # manage per-environment settings
bmo config auth                          # switch auth mode or update credentials
bmo config package-store                 # update AccountUri / ContainerName
bmo config servicebus                    # configure Service Bus transport (opt-in)

Day-to-day workflows#

Global flags#

These are available on every command:

Flag	Purpose
`--verbose` / `-v`	Route phase progress and full error/stack-trace detail to the console (normally only logged to file). Use when troubleshooting.
`--yes` / `-y`	Skip confirmation prompts (for automation). Destructive commands confirm interactively unless this is set; non-interactive sessions never prompt.
`--quiet` / `-q`	Disable progress spinners/animations. Errors and structured output are unaffected.
`--version`	Print the `bmo` version.

Exit codes (consistent across commands): 0 success · 1 operational failure · 2 invalid arguments/usage.

Action commands deploy, manage remove, manage uninstall, and sync-agent also accept -o json for machine-readable per-server/per-package results.

Preflight check#

Verify connectivity before deploying — Command Center reachability (and auth), every configured agent, and package-store configuration:

bmo doctor                # human table; exits non-zero if any check fails
bmo doctor -o json        # machine-readable, for CI

Deploy a version#

bmo deploy <ENV> <VERSION> [ZIP_PATH] [--server <name>] [--via http|blob] [--archive] [--no-start]

Argument / flag	Default	Notes
`ENV`	—	Target environment (`Acceptatie` or `Productie`)
`VERSION`	—	Version string matching the package (e.g. `25.5.29`)
`ZIP_PATH`	—	Path to the deployment zip. Required for `--via http`.
`--server <name>`	all configured agents	Restrict to one or more named agents
`--via`	`http`	`http`: Orchestrator pushes the zip to the Agent. `blob`: Agent pulls from the package store.
`--archive`	off	Copy the zip into blob storage after an `http` deploy
`--no-start`	off	Install or update the Windows Service but leave it stopped. The service's `StartMode` stays `Automatic (Delayed)` so the next reboot will start it normally. Use this for maintenance windows or coordinated multi-host rollouts.

--via http — use this when you have a zip locally and want to deploy directly without going through the package store. Useful for development and hotfixes.

--via blob — use this for production deploys. The package must already be in the store (pushed via bmo package push or your CI pipeline). Environments with RequiresAcceptedStatus: true require the package to be in Accepted status before bmo will proceed.

--no-start caveat: While the service is stopped the Agent's state monitor will emit a Warning-level HealthAlert log every monitoring interval (default 5 min). This is expected and resolves once the service is manually started.

Promote a package: UAT → Production#

# 1. Review what is waiting in UAT
bmo package list --status Uat

# 2. Inspect the package manifest
bmo package show 25.5.29

# 3. Promote to Accepted
bmo package accept 25.5.29

# 4. Deploy to production
bmo deploy Productie 25.5.29 --via blob

Reject a package (marks it terminal; no further deploys possible):

bmo package reject 25.5.29 --reason "Regression in invoicing module"

Supersede an older version when it has been replaced:

bmo package supersede 25.5.28 --by 25.5.29

Delete / prune packages from the store (blob + manifest). delete removes one version; prune bulk-removes by status and is a dry run unless --apply:

bmo package delete 25.5.28                      # confirmation prompt (skip with --yes)
bmo package prune --status Rejected             # dry run: list what would be deleted
bmo package prune --status Superseded --apply   # actually delete

Bring up a new Agent server#

Replay all Accepted packages onto a freshly installed server in a single command:

bmo sync-agent Productie vmbenefitsbmp2

Inspect environment state#

bmo query status                                       # installations on all agents (all environments)
bmo query status Productie                              # optional positional: filter to one environment
bmo query status --server vmbenefitsbmp1               # single agent
bmo query status --match 25.5.*                        # narrow to a version family
bmo query status --match 25.5.10..25.5.29 --strict     # CI gate: exit 1 if any matched component is not Running
bmo query customers Productie                          # list all customers
bmo query stages Productie --customer "BCS"            # stages for one customer
bmo query versions Productie                           # versions not yet assigned to any stage
bmo query installed Productie --match 25.5.* --health  # version rows with aggregated health column

All query commands accept:

-o table (default) | -o json | -o md — output format
--query <jmespath> — filter or reshape JSON output

`--match` grammar (shared by `query status`, `query installed`, and `manage uninstall`)#

Shape	Interpretation	Example
`/.../`	Regex (case-insensitive)	`/^25\.[5-6]\..+/`
Contains `..`	Semver range (inclusive bounds; either side may be empty)	`25.5.10..25.5.29`, `..25.5.29`, `25.5.10..`
Contains `*` / `?`	Glob	`25.5.*`
Otherwise	Exact match	`25.5.29`

Bare * matches all versions.

`query status --strict` for CI#

--strict exits non-zero (1) if any matched component is not Running. Combine with --match to gate a release; use it bare to gate the whole fleet. Exit codes:

Code	Meaning
`0`	All matched components Running (or no `--strict`, no unreachable servers)
`1`	At least one matched component not Running, or at least one server unreachable
`2`	Invalid `--match` pattern

JSON output remains clean (no extra warning text on stdout); the exit code is the only out-of-band signal in -o json mode.

`query installed --health`#

Off by default. When set, the command adds an aggregated Health column (Running / Partial / Stopped / Unreachable / Unknown) by fanning out one GetAgentInstallations call per distinct server. JSON/Markdown shape is unchanged when --health is absent — existing scripts are not affected.

Deploy history#

Read past version transitions for an environment (newest first) from the Command Center deploy log, with foreign keys resolved to customer / stage / old→new version:

bmo logs Productie                                  # most recent 100 entries
bmo logs Productie --customer "BCS" --since 2026-06-01
bmo logs Productie --limit 20 -o json               # machine-readable; supports --query

Rollbacks / downgrades — why there is no `bmo rollback`#

There is intentionally no rollback command. A customer-stage version change is not just a binary swap: upgrades run forward database migrations driven by the stage's scheduled tasks (TblToolSchedule), and those migrations are generally one-way. Re-deploying older binaries would leave the schema ahead of the application — an unsafe state. Recovery is therefore deliberate:

Bad binary, schema unchanged: redeploy a prior build explicitly with bmo deploy <ENV> <oldVersion>.
A true downgrade is a migration-aware operational procedure (schema + scheduled jobs + data), not a one-shot CLI action.

Use bmo logs to inspect what changed and decide deliberately.

Lifecycle and cleanup#

# Dry-run: show what would be removed
bmo manage lifecycle Productie --older-than-days 90

# Apply the cleanup
bmo manage lifecycle Productie --older-than-days 90 --apply

# Remove a specific version from all agents in an environment
bmo manage remove Productie 25.4.10

Version cleanup#

Use bmo query installed <ENV> --unused to see all versions installed on agents that are no longer assigned to any active customer stage. Then use bmo manage uninstall <ENV> --match <PATTERN> --apply to remove them.

Dry-run first, then apply:

bmo query installed Acceptatie --unused
bmo manage uninstall Acceptatie --match 25.5.* --apply --yes

Windows Service control#

bmo manage service Productie 25.5.29 Start
bmo manage service Productie 25.5.29 Stop
bmo manage service Productie 25.5.29 Restart

Service resource management#

Adjust CPU priority and affinity on a running Windows Service without RDP access:

bmo manage service-resources <ENV> <VERSION>
    [--priority <Idle|BelowNormal|Normal|AboveNormal|High>]
    [--affinity <spec>]
    [--clear-priority]
    [--server <name>]

Flag	Description
`--priority`	Set process priority. Persisted to the IFEO registry — survives `services.msc` restarts and reboots. `RealTime` is not exposed.
`--affinity`	CPU core affinity. Accepts comma-list (`0,1,2`), range (`0-3`), hex bitmask (`0xF`), or decimal core index (`15`). Apply-once — lost when the service restarts (Windows limitation).
`--clear-priority`	Remove the IFEO registry priority override, restoring the OS default on the next service start.
`--server`	Target specific server(s). Defaults to all configured agents.

Examples:

# Reduce UAT service to BelowNormal priority on all agents
bmo manage service-resources Acceptatie 25.5.29 --priority BelowNormal

# Pin to cores 0 and 1 only
bmo manage service-resources Acceptatie 25.5.29 --affinity 0,1

# Combined: lower priority and pin to a range
bmo manage service-resources Acceptatie 25.5.29 --priority BelowNormal --affinity 0-3

# Remove the priority registry override
bmo manage service-resources Acceptatie 25.5.29 --clear-priority

Persistence model:

Setting	Survives `services.msc` restart?	Survives reboot?	Survives version redeploy?
Priority (via IFEO registry)	Yes	Yes	No — new version = new executable path. Re-apply after redeploy.
Affinity (live process only)	No	No	No

The service must be running when service-resources is called. Targeting a stopped service returns an error.

IIS pool management#

bmo manage iis Productie_25.5 Recycle
bmo manage iis Productie_25.5 Start
bmo manage iis Productie_25.5 Stop

Customer data#

# Export all customer/stage data for an environment
bmo manage customer-data export Productie C:\Backup\productie.bmpkg

# Export a single customer
bmo manage customer-data export Productie C:\Backup\bcs.bmpkg --customer "BCS"

# Import into another environment
bmo manage customer-data import Acceptatie C:\Backup\bcs.bmpkg

# Inspect a package before importing
bmo manage customer-data show C:\Backup\bcs.bmpkg

# --verbose also scans log-table date ranges and prints a table of the active &
# upcoming tool schedules in the package (Key / Stage / Recurrence / Effective
# window / ToolCode / LastRun) so they can be vetted before import. "Active &
# upcoming" = Active=true and not past EndEffective; future-dated one-shots are
# included because they fire on the target after import.
bmo manage customer-data show C:\Backup\bcs.bmpkg --verbose

# Remove a stage
bmo manage remove-stage Productie <stage-id>

Certificates#

bmo manage certificate list     --server vmbenefitsbmp1
bmo manage certificate download --server vmbenefitsbmp1
bmo manage certificate upload   --server vmbenefitsbmp1
bmo manage certificate delete   --server vmbenefitsbmp1

Configuration reference#

`Orchestrator.*`#

Setting	Default	Description
`Agents`	`{}`	Named Agent endpoints: `{ name: { BaseUrl, UseSasForBlob } }`
`ApiKey`	`""`	Shared secret sent as `X-Api-Key` to Agents when `AuthMode = ApiKey`
`AuthMode`	`ApiKey`	`ApiKey` \| `mTLS` \| `EntraId` — for outbound Agent calls
`PollingIntervalSeconds`	`3`	Frequency of job status polls against the Agent
`PollingTimeoutMinutes`	`10`	Maximum wait for a deploy job to complete
`VersionsPath`	`F:\Versions`	Local directory used as the default zip source
`Environments.<Env>.RequiresAcceptedStatus`	`false`	Reject `--via blob` deploys unless the package status is `Accepted`

`Orchestrator.CommandCenter.*`#

Setting	Default	Description
`BaseUrl`	`""`	HTTP base URL of the Command Center service, e.g. `https://cc.internal:8090`
`AuthMode`	`ApiKey`	`ApiKey` \| `mTLS` \| `EntraId` — for outbound Command Center calls
`ApiKey`	`""`	API key sent to Command Center when `AuthMode = ApiKey`
`ClientCertificate.*`	—	Certificate settings for `AuthMode = mTLS` (same shape as `ClientCertificate.*` below)
`EntraId.TargetScope`	`""`	App Registration scope for `AuthMode = EntraId`

`ClientCertificate.*` (for Agent `AuthMode = mTLS`)#

Setting	Default	Description
`Source`	`Store`	`File` \| `Store` \| `KeyVault`
`StoreLocation`	`LocalMachine`	Windows cert store location (`Store` source)
`StoreName`	`My`	Windows cert store name (`Store` source)
`Thumbprint`	`""`	Certificate thumbprint (`Store` source)
`FilePath` / `Password`	`""`	PFX file path and password (`File` source)
`KeyVaultUrl` / `CertificateName`	`""`	Key Vault URL and certificate name (`KeyVault` source)

`EntraId.*` (for Agent `AuthMode = EntraId`)#

Setting	Description
`TargetScope`	App Registration scope, e.g. `api://<ClientId>/.default`

`PackageStore.*`#

Setting	Default	Description
`AccountUri`	`""`	Azure Blob Storage account URI
`ContainerName`	`deployments`	Blob container holding manifests and zip files
`ConnectionStrings.PackageStore`	—	Full Azure Storage connection string (alternative to `AccountUri`)

Auth modes for outbound calls to Agents#

Mode	How it works	What to configure
`ApiKey` (default)	Sends `X-Api-Key` header	`Orchestrator.ApiKey` — must match `Authentication.ApiKey` on the Agent
`mTLS`	Client certificate in TLS handshake	`ClientCertificate.*` — thumbprint must be in the Agent's `ClientAuthentication.AllowedThumbprints`
`EntraId`	Acquires bearer token via MSAL	`EntraId.TargetScope` — must match the audience configured on the Agent's `AzureAd` section

The same auth modes apply to Command Center calls via Orchestrator.CommandCenter.AuthMode.

Service Bus transport (optional)#

By default bmo communicates with Agents over HTTP. To switch to scatter-gather via Azure Service Bus, add a ServiceBus section to appsettings.json or run bmo config servicebus:

"Orchestrator": {
  "ServiceBus": {
    "Namespace": "my-namespace.servicebus.windows.net",
    "SnapshotContainerUrl": "https://...",
    "CommandsTopic": "deploy.commands.v1",
    "RepliesTopic": "deploy.replies.v1"
  }
}

When ServiceBus.Namespace is set it replaces the HTTP transport for deploy operations. Each Agent must also have Agent.ServiceBus.Namespace configured to match. See agent-setup-and-operations.md for the Agent-side config.

Troubleshooting#

Symptom	Likely cause	Fix
`ZIP_PATH is required`	Using `--via http` without a zip argument	Add the path or switch to `--via blob`
`401 / 403` calling Agent	Auth mismatch	Compare `bmo config show` with `Authentication.ApiKey` on the Agent
`401 / 403` calling Command Center	Auth mismatch	Compare `Orchestrator.CommandCenter.ApiKey` in `bmo config show` with `ClientAuthentication.ApiKey` on the Command Center
Deploy blocked on `RequiresAcceptedStatus`	Package not promoted yet	Run `bmo package accept <version>` first
Agent not in `bmo query status`	Wrong URL or Agent unreachable	Check `Orchestrator.Agents.<name>.BaseUrl`; test with `curl http://<host>:5100/health`
Command Center unreachable	Service stopped or wrong URL	Check `Get-Service Benefits-Mgmt-CommandCenter` on the CC host. Verify `Orchestrator.CommandCenter.BaseUrl`.
`bmo` cannot find `appsettings.json`	Running `bmo` from wrong directory	Run from the directory that contains `bmo.exe`
Certificate load error	Cert not in store or wrong thumbprint	Verify with `Get-ChildItem Cert:\LocalMachine\My \\| Where-Object Thumbprint -eq '...'`
`service-resources` returns "service not running"	Targeting a stopped service	Start the service first with `bmo manage service <ENV> <VERSION> Start`, then apply resource settings

Operator checklist#

[ ] bmo config init completed without errors
[ ] bmo config show reviewed — all agents, environments, Command Center URL, and credentials correct
[ ] Command Center health check passes: curl http://<cc-host>:8090/healthz returns Healthy
[ ] bmo query status returns health for all expected agents
[ ] bmo package list returns results (package store connected)
[ ] bmo deploy Acceptatie <version> <zip> --via http succeeds on a test package
[ ] bmo manage lifecycle Acceptatie --older-than-days 90 (dry-run) completes without errors
[ ] For production: bmo package accept <version> → bmo deploy Productie <version> --via blob succeeds
[ ] ApiKey stored securely — not committed to source control

Benefits Orchestrator (bmo) — Setup and Operations#