Skip to content

Benefits Command Center — Setup and Operations#

The Command Center is a standalone Windows Service that owns all SQL access for the Benefits system. In v2.0.0 all direct database calls were moved out of bmo and into this service; bmo now communicates with it over HTTP.

What is the Command Center?#

The Command Center:

  • Maintains the deployment registry (application versions, customer stages)
  • Runs SQL pre-checks and finalization steps during deploys
  • Exposes an HTTP API consumed by bmo and (when configured) by Azure Service Bus event consumers

Every bmo deploy call touches the Command Center. It must be reachable before bmo can complete a deployment.

Prerequisites#

  • Windows Server (x64)
  • SQL Server accessible from the host where Command Center runs
  • .NET 10 Runtime (or higher)
  • Inbound firewall rule on port 8090 (or your configured port) from bmo workstations and CI runners
  • A server certificate if you want HTTPS or mTLS

Installation#

Publish or copy the Benefits.CommandCenter binaries to a local directory (e.g. C:\Benefits\CommandCenter), then run the install script from that directory:

cd C:\Benefits\CommandCenter
.\Install.ps1

The script registers and starts the Benefits-Mgmt-CommandCenter service with StartupType = Automatic.

Start, stop, restart:

Start-Service   Benefits-Mgmt-CommandCenter
Stop-Service    Benefits-Mgmt-CommandCenter
Restart-Service Benefits-Mgmt-CommandCenter

The service runs under LocalSystem by default. Change the service identity in the Services MMC snap-in if your SQL account requires a named service account.

Restart the service after any configuration change.

Configuration#

All settings live in appsettings.json (and appsettings.Production.json for production overrides) next to the executable.

SQL connection strings#

"ConnectionStrings": {
  "CommandCenter_Productie": "Server=sqlserver;Database=BM_Productie;Integrated Security=true;",
  "CommandCenter_Acceptatie": "Server=sqlserver;Database=BM_Acceptatie;Integrated Security=true;"
}

Each key must be CommandCenter_<environment> where <environment> matches the environment name used in bmo deploy (e.g. Productie, Acceptatie).

Port#

"CommandCenter": {
  "Port": 8090
}

Default is 8090. If you change this, update bmo's CommandCenter.BaseUrl to match.

Client authentication#

"CommandCenter": {
  "ClientAuthentication": {
    "ApiKey": "your-secret-key",
    "AllowedThumbprints": []
  }
}
Setting Description
ApiKey Shared secret that bmo sends as X-Api-Key. Must match bmo's CommandCenter.ApiKey. Leave empty to disable API key auth.
AllowedThumbprints Client certificate thumbprints allowed for mTLS. Empty = any valid cert is accepted.

Server certificate (HTTPS / mTLS)#

"CommandCenter": {
  "ServerCertificate": {
    "Source": "Store",
    "StoreLocation": "LocalMachine",
    "StoreName": "My",
    "Thumbprint": "ABCD1234..."
  }
}
Source What to set
Store (default) StoreLocation, StoreName, Thumbprint
File FilePath, Password

If Thumbprint is empty the service starts on plain HTTP.

Entra ID (optional)#

"AzureAd": {
  "Instance": "https://login.microsoftonline.com/",
  "TenantId": "<tenant-id>",
  "ClientId": "<app-registration-client-id>"
}

When AzureAd.ClientId is non-empty, Entra ID bearer token authentication is enabled alongside ApiKey and mTLS. bmo must be configured with CommandCenter.AuthMode = EntraId and the matching TargetScope.

Service Bus (optional)#

"CommandCenter": {
  "ServiceBus": {
    "Namespace": "my-namespace.servicebus.windows.net",
    "RepliesTopic": "deploy.replies.v1",
    "StateTopic": "agent.state.v1",
    "PackageEventsTopic": "bmo.package.events.v1",
    "DeployCommandsTopic": "bmo.deploy.commands.v1",
    "AgentCommandsTopic": "agent.commands.v1",
    "AgentRepliesTopic": "agent.replies.v1",
    "SnapshotContainerUrl": "https://...",
    "MaxAgentStateAge": "00:15:00",
    "AgentTimeouts": {
      "Status": "00:00:05",
      "ServiceAction": "00:01:00",
      "CertificateBinary": "00:02:00",
      "Default": "00:00:30"
    }
  }
}

When Namespace is set, the Command Center:

  • Subscribes to deploy reply events and agent state snapshots from Azure Service Bus.
  • Routes all operator commands (manage service, manage iis, manage certificate, query status, etc.) to agents via Service Bus instead of direct HTTP. Agents no longer need an inbound port reachable from the Command Center host — only outbound Service Bus connectivity is required on agent hosts.

The AgentCommandsTopic (agent.commands.v1) carries operator commands to agents. The AgentRepliesTopic (agent.replies.v1) carries replies back. Each request uses an ephemeral per-correlationId subscription that is created before the command is sent and deleted after the reply is received.

MaxAgentStateAge controls how stale an agent's last heartbeat can be before the Command Center returns 503 Service Unavailable without sending a Service Bus message (default: 15 minutes).

Leave Namespace empty for HTTP-only deployments. In that mode the Command Center calls each agent's HTTP endpoint directly (Spec A behaviour), and agents must have port 5100 (or their configured port) reachable from the Command Center.

When ServiceBus.Namespace is configured on both the Command Center and each agent, the network diagram simplifies to:

bmo ──HTTPS──► CC ──SB──► agent (outbound only)
               CC ◄──SB── agent

Agent host firewall in SB-only mode: - Remove inbound TCP/5100 rule from the Command Center host. - Keep inbound TCP/5100 open only from the local subnet (for the management web UI) or lock it down completely if the UI is not used. - Add outbound HTTPS (port 443) to <your-namespace>.servicebus.windows.net.

The agent's HTTP server (Port 5100) remains active and serves the local management dashboard and direct deployments; it simply has no inbound route from the Command Center.

Configuration reference#

Setting Default Description
CommandCenter.Port 8090 Kestrel listen port
CommandCenter.MaxRequestBodySizeBytes 524288000 Max request body size (bytes)
CommandCenter.ClientAuthentication.ApiKey "" Inbound API key for bmo clients
CommandCenter.ClientAuthentication.AllowedThumbprints [] Client cert thumbprints for mTLS
CommandCenter.ServerCertificate.Source Store Certificate source: Store | File
CommandCenter.ServerCertificate.Thumbprint "" Server cert thumbprint (Store source)
CommandCenter.ServiceBus.Namespace "" Service Bus FQDN; empty = HTTP-only mode
CommandCenter.ServiceBus.AgentCommandsTopic agent.commands.v1 Topic CC publishes operator commands to
CommandCenter.ServiceBus.AgentRepliesTopic agent.replies.v1 Topic CC listens for agent replies on
CommandCenter.ServiceBus.MaxAgentStateAge 00:15:00 Stale-heartbeat threshold for fast-fail 503
CommandCenter.ServiceBus.AgentTimeouts.Default 00:00:30 Reply wait timeout for most operations
CommandCenter.ServiceBus.AgentTimeouts.CertificateBinary 00:02:00 Reply wait timeout for cert upload/download
ConnectionStrings.CommandCenter_<Env> — SQL connection string per environment

Health check#

curl http://localhost:8090/healthz

Returns 200 OK with body Healthy when the service is running.

Stage removal (background job)#

bmo manage stage remove enqueues a background job and polls for completion. This means:

  • The HTTP response to POST /api/customers/stages/cleanup is 202 Accepted immediately, with a { jobId, statusUrl } body.
  • GET /api/customers/stages/cleanup/{jobId} returns the current phase and rows-deleted count, polling-friendly.
  • The batch deletion runs against the worker's lifetime token — it continues even after the bmo process exits (for example, if you kill the CLI and re-run it with the same job ID).
  • In-memory job state is lost on CC restart. If the CC is restarted mid-deletion, simply re-run the bmo command; the DELETE is idempotent (WHERE CustomerStageKey = @StageKey skips already-deleted rows).

Status phases: Queued → LookingUpStage → DeletingServerLogs → NullifyingFks → DeletingStage → Completed Force path: Queued → LookingUpStage → Deactivating → Deactivated

Troubleshooting#

Symptom Likely cause Fix
bmo deploy fails with connection error to Command Center Service stopped or wrong URL Check Get-Service Benefits-Mgmt-CommandCenter. Verify bmo's CommandCenter.BaseUrl.
401 / 403 from Command Center API key mismatch Compare CommandCenter.ClientAuthentication.ApiKey (server) with bmo's CommandCenter.ApiKey (client).
SQL errors in Command Center logs Wrong connection string or SQL server unreachable Check ConnectionStrings.CommandCenter_<Env> in appsettings.json. Verify SQL server accessibility from the Command Center host.
Service fails to start Port conflict or certificate not found Check Windows Event Log. For cert errors, verify Thumbprint against Get-ChildItem Cert:\LocalMachine\My. For port conflicts, change CommandCenter.Port.
Stage removal stuck in DeletingServerLogs forever Very large TblServerLog table or slow SQL Inspect CC logs for batch progress (deleted N TblServerLog rows). Let it run — each 1000-row batch is auto-committed. Restarting CC is safe; re-run the command after restart.

Operator checklist#

  • [ ] Service installed and running: Get-Service Benefits-Mgmt-CommandCenter shows Running
  • [ ] Health check passes: curl http://localhost:8090/healthz returns Healthy
  • [ ] SQL connection strings configured for each environment
  • [ ] ClientAuthentication.ApiKey set and matches bmo's CommandCenter.ApiKey
  • [ ] Port 8090 (or configured port) open inbound from bmo workstations and CI runners
  • [ ] Server certificate configured if HTTPS is required