Shared configuration#

CommandCenter's services read their settings from Azure App Configuration, with secrets in Azure Key Vault. A change made there reaches every instance of every service within about half a minute — without anyone signing in to a server or editing a file. The /config settings pages show what the store holds and change it (below); what they do not cover yet is edited in the portal or with the CLI.

The settings pages on `/config`#

flowchart LR
    SPA["/config/settings/{page}"] -- "GET /api/settings/pages/{page}" --> WebApi
    SPA -- "POST /bff/authz/decisions (config.read)" --> BFF["Gateway"] --> OPA
    WebApi -- "config.read" --> OPA
    WebApi -- "key-values by label (App Configuration Data Reader)" --> AppConfig["App Configuration"]
    Catalogue["SettingsCatalogue (code)"] -. "the allow-list" .-> WebApi

Hold "Alt" / "Option" to enable pan & zoom

The catalogue is the allow-list (Shared.Configuration/Settings/SettingsCatalogue.cs): each page lists its settings with their key, kind, whether a change applies at once or after a restart, and whether it is a secret. A key that is not in the catalogue is not shown, and will not be writable. Today: Cross-origin callers (platform), Log Analytics, Service Bus, Customer-data storage, Package store (per environment).
Read from the store, not from the host's configuration, so every label is visible: the value a host of that scope reads (the most specific label wins), the labels it overrides, when it changed and — once the pages can write — who changed it.
A secret's value never leaves. A secret shows as set, in Key Vault: {secret name}. A secret stored as a plain value is flagged: move it to Key Vault and keep a Key Vault reference.
Everything needs config.read (401 / 403 / 503 as elsewhere). Without an App Configuration store the pages say so (503 not-configured).

Changing a setting#

sequenceDiagram
    participant Op as Operator
    participant SPA as /config/settings/{page}
    participant API as WebApi
    participant OPA
    participant AC as App Configuration
    participant Hosts as Every host
    Op->>SPA: Change, Save
    SPA->>API: PUT /api/settings/pages/{page}, the version read
    API->>API: every value checked (400)
    API->>OPA: config.write per key (+ secrets.write for a secret)
    API->>AC: versions still the ones read? (else 409, nothing written)
    API->>AC: write at the page's label, tagged cc-changed-by
    API->>AC: move the sentinel once
    AC-->>Hosts: reload within a refresh
    API-->>SPA: the page as it now is

Hold "Alt" / "Option" to enable pan & zoom

At the page's label only: platform for a platform page, {env} for an environment page. A service's own label ({env}/webapi) and the baseline are shown but not edited; when one of them overrides the page's label, the editor says that a change there would have no effect.
Everything first, then the writes: a wrong value (400, per key), a refused key (403, with the policy's reasons) or a stale version (409, "changed by someone else") writes nothing. The version is App Configuration's own ETag; a list has one version over its items.
Secrets by name only: a secret is set by naming a secret in the host's Key Vault (Configuration:KeyVault:Uri); App Configuration then keeps a Key Vault reference. Putting a secret value into Key Vault is not done from these pages.
Which secrets are referred to, and when they expire: /config/credentials lists every Key Vault reference at the platform's labels and this environment's (and at no label), with the secret's metadata from the host's vault: enabled, last change, expiry. It warns 30 and 14 days before a secret expires, and flags one that is missing, disabled or expired, or that a host instance could not resolve (from host status). A reference to another vault shows as not readable there. It reads metadata only, never a value; the host's identity needs Key Vault Secrets User (or Officer) on its vault for it.
Remove here deletes the setting at the page's label, so the inherited value applies again.
One reload: the scope's sentinel moves once per save (Settings:Sentinel for the environment, Settings:PlatformSentinel for the platform), so a save of several settings applies together.
Audit: every change is an audit event (EventIds 3180 changed, 3181 removed) with who, the key and the label, never the value.
Who may write the store: the identity of the host that serves the page needs App Configuration Data Owner. That is the WebApi serving the platform: the platform host, or a WebApi in the default combined role (see WebApi's roles). A host with Data Reader only answers "could not be written" and nothing is saved. Rotating the gateway's signing key on /config/credentials also writes Key Vault, so that host needs Key Vault Secrets Officer there (docs/authorization.md).

What each host reads#

flowchart LR
    subgraph local ["On the VM (set once at deployment)"]
        Boot["Configuration:*<br/>endpoint, identity, environment"]
        Env["environment variables<br/>(break-glass, reported)"]
    end
    subgraph azure ["Azure"]
        AppConfig["App Configuration<br/>labels: platform, {env}, {env}/{service}"]
        KeyVault["Key Vault<br/>secrets"]
        Status["Blob container config-status"]
    end
    Host["Service instance"]

    Boot --> Host
    AppConfig -- "settings, refreshed on the sentinel" --> Host
    KeyVault -- "Key Vault references" --> Host
    Env -- "wins over App Configuration" --> Host
    Host -- "what it loaded, every minute" --> Status

Hold "Alt" / "Option" to enable pan & zoom

Bootstrap, on the VM (Configuration:*): only what a host needs to find the store — its endpoint, the identity to use, the environment name. Nothing else lives in .env or appsettings.json on a server.
Labels, later wins: no label → platform → {env} → {env}/{service} (a host serving every environment, like the gateway: no label → platform → platform/{service}). Keys keep the paths the code reads (LogAnalytics:WorkspaceId); the label decides which hosts get a value.
Order: appsettings.json → App Configuration → environment variables → command line. An environment variable on a VM still wins — the per-server break-glass — and is reported as a shadowed key, so a change in the UI that seems to have no effect can be explained.
Secrets are Key Vault references in App Configuration, resolved with the host's identity, from the configured vault only. A reference that cannot be read becomes an empty value and is reported; it never stops a host.

When a change applies#

sequenceDiagram
    participant Operator
    participant AppConfig as App Configuration
    participant Host as Every instance
    participant Status as config-status
    Operator->>AppConfig: change settings (one or several)
    Operator->>AppConfig: bump the sentinel
    loop every 30 s
        Host->>AppConfig: did a sentinel move?
    end
    AppConfig-->>Host: yes: reload all settings at once
    Host->>Status: revision 0/7, restart pending: ConnectionStrings:…

Hold "Alt" / "Option" to enable pan & zoom

Nothing reloads until a sentinel moves: Settings:Sentinel (label {env}) for an environment, Settings:PlatformSentinel (label platform) for the platform. A batch of changes therefore applies together.
A host reloads all its settings when a sentinel moves. Whether a setting takes effect at once depends on the code that reads it; connection strings, sign-in (Authentication, AzureAd) and ports (Kestrel, Urls) do not, and the host reports them as restart pending.
The revision a host reports is its sentinels' values (platform/environment, e.g. 3/7).

What applies without a restart#

Setting	On the next request
`LogAnalytics:WorkspaceId`	The Exceptions page's Log Analytics source and its feature switch read it per query.
`CustomerData:Storage:*` (`ServiceUri`, `ConnectionString`, `Container`)	Customer-data export and import use a store rebuilt for the new settings.
`BmoBlob:ServiceUri`, `ConnectionString`, `Container`	The package store behind the package reference is rebuilt. (`BmoBlob:SasTtl` still needs a restart.)
`Agents:ServiceBus:*`	Heartbeats and agent commands move to the new namespace or topics within 15 s. (`MaxAgentStateAge` as the per-agent router reads it still needs a restart.)
`Platform:SqlServers:*`	Stage connections resolve through the new registry (`docs/sql-servers.md`).
`Platform:Environments:*`	The gateway serves the new catalogue (`docs/environments.md`).
`Logging:Runtime`	Log levels change and revert at their expiry (`docs/logging.md`).

A client is rebuilt only when the settings it was built from change, so a refresh that changes something else keeps it warm. The one it replaces is kept for two minutes, so a request already using it finishes on it. Every Azure client signs in as the host: the managed identity, or the alternate identity in Configuration:Identity.

Host status#

Every instance writes {env}/{service}/{instance}.json (or platform/{service}/…) to the config-status blob container every minute and right after a change:

{ "service": "webapi", "environment": "prod", "instance": "vm-cc-01", "source": "AppConfiguration",
  "revision": "3/7", "startupRevision": "3/5", "restartPendingKeys": ["ConnectionStrings:CommandCenter"],
  "shadowedKeys": [], "unresolvedSecrets": [] }

Only key names — never values.

The Hosts panel (`/config/hosts`)#

flowchart LR
    SPA["/config/hosts"] -- "GET /api/hosts/status (config.read)" --> WebApi
    WebApi -- "list {env}/ and platform/" --> Status[("config-status blobs")]
    SPA -- "POST /api/hosts/{scope}/{service}/restart (hosts.restart)" --> WebApi
    WebApi -- "Platform:Hosts:{scope}:{service}:Restart at label platform,<br/>then Settings:PlatformSentinel" --> AC["App Configuration"]
    AC -. "reload" .-> Hosts["every instance of that service"]
    Hosts -- "lease {scope}/{service}/restart.lock" --> Status
    SPA -- "POST /api/hosts/migrate (hosts.migrate)" --> WebApi --> DB[("this environment's database")]

Hold "Alt" / "Option" to enable pan & zoom

Instances of this environment and of the platform, grouped by service, as each last reported: the revision it runs (and the one it started on), where its settings came from (App Configuration, a snapshot because the store was down at start, local files, or an unreachable store), the settings that wait for a restart, the keys an environment variable on the host overrides, and Key Vault references it could not read. An instance with no report for three status intervals is marked; it may be gone.
Restart (permission hosts.restart) asks every instance of one service to restart, with a reason. The request is a setting at label platform (Platform:Hosts:{scope}:{service}:Restart, { RequestedAt, RequestedBy, Reason }) and the platform sentinel moves, so every host sees it within a refresh. An instance that started after the request has nothing to do. The others take turns: each takes a 60 s blob lease on {scope}/{service}/restart.lock, logs the audit event 928 and stops gracefully; Docker (restart: unless-stopped) starts it again. The next instance can take the lease only once it runs out, which gives the first one time to come back. The panel shows the instance return with a new start time. Audit 3182 records who asked.
Where nothing would start a host again — the development stack, because dotnet watch does not exit when the app stops — Configuration__Restart__Enabled=false; the host logs 929 once and keeps running, and the panel offers no Restart for that service.
Apply migrations (permission hosts.migrate) applies this environment's pending WebApi migrations and lists exactly what it applied (or "nothing to apply"). It is the same work as GET /api/maintenance/migrate; audit 3183.
Without a status container (Configuration:Status) there is nothing to show and a restart has no lease: every instance stops at once (logged as 934).

Setting	Default
`Configuration__Restart__Enabled`	`true`	act on restart requests
`Configuration__Restart__CheckInterval`	`00:00:15`	how often a host looks for one
`Configuration__Restart__LeaseDuration`	`00:01:00`	how long one instance's turn lasts

Asking a restart writes App Configuration, so the host that serves the panel needs App Configuration Data Owner (as for the settings pages); reading status needs Storage Blob Data Reader on config-status, which the hosts' Contributor role already covers.

WebApi's roles (`Hosting:Role`)#

WebApi can serve one environment, the platform, or both. The role is set locally, like Configuration:*, because it decides which labels the host reads.

`Hosting:Role`	Serves	Reads	Runs
`combined` (default)	everything, as WebApi always did	none → `platform` → `{env}` → `{env}/webapi`	everything
`environment`	the environment's endpoints: customers, analytics, jobs, tasks, the servers and which agent manages each, features, exceptions, the BMO slices	as `combined`	jobs, the task queue, the `CommandCenter` database, operator commands; reads the agent registry from the platform host (below)
`platform`	the platform's: environments, access and OPA's bundle, credentials, the identity provider, the agent registry and agent settings, the Hosts panel and restart requests, the SQL Server registry, platform settings pages, platform log-level targets	none → `platform` → `platform/webapi` (the platform sentinel only)	the platform database and the agent heartbeat consumer; no jobs, no task queue, no `CommandCenter` database

The agent registry is in the platform database (#363): a SQLite file that only the host serving the platform (combined or platform) opens and migrates at start. Exactly one instance per deployment may serve the platform. Environment hosts read the registry from it over HTTP (#364): Agents:Registry:Url (the platform host's address; without it an environment host fails at start) and Agents:Registry:Token (the same on the platform host). See agents.

Pages that span both are split by what each route touches: - SQL Servers: the registry is the platform's; the list of the servers this environment's stages use, and the test, which signs in from the environment's backend, stay with the environment. - Hosts: the panel and restart requests are the platform's; Apply migrations stays with the environment. - Settings pages and log levels: split by scope. A page or target outside this host's scope answers 404 or "not a target".

On a platform host: - The Hosts panel, the Key Vault inventory and the SQL Server registry show the environment named in ?environment=, when the catalogue has it. Without one they show the platform only. - A restart can be requested for any environment in the catalogue.

How the platform pages reach it. - The SPA calls them through /platform/api (services/web-api/platform.ts), never environment-prefixed. The gateway's api-platform route sends that to cluster-platform as /api/..., with the user token. - Until a deployment runs a separate platform host, cluster-platform points at the environment's WebApi, which runs combined, so nothing changes. A deployment with a platform host points it there (ReverseProxy__Clusters__cluster-platform__Destinations__platform__Address) and, once environment hosts read the agent registry over HTTP, sets Hosting__Role=environment on the others. - Calls are routed per route, not per page: - agent settings go to the platform, Restart to the environment; - the SQL Server registry goes to the platform, the stage servers and the test to the environment; - the Hosts panel and restart requests go to the platform, Apply migrations to the environment; - a settings page goes where its catalogue scope says; - log levels are read from both hosts and written where the target label belongs: platform, platform/* on the platform host.

Running a platform host#

flowchart LR
    Browser -- "/platform/api/…" --> BFF["Gateway<br/>route api-platform"]
    Browser -- "/env/{env}/api/…" --> BFF
    BFF -- "cluster-platform" --> Platform["commandcenter-webapi-platform<br/>Hosting:Role platform"]
    BFF -- "cluster-next, tagged per environment" --> Env["commandcenter-webapi<br/>Hosting:Role environment"]
    Platform -- "writes platform labels<br/>(Data Owner)" --> AC["App Configuration"]
    Platform -- "signing keys<br/>(Secrets Officer)" --> KV["Key Vault"]
    Env -- "reads (Data Reader)" --> AC
    OPA -- "access bundle" --> Platform
    Env --> DB[("CommandCenter database")]

Hold "Alt" / "Option" to enable pan & zoom

Compose has the service behind the profile platform. Switch it on together with the other three settings (.env.template lists them):

Setting	Value	What it does
`COMPOSE_PROFILES`	`platform`	starts `commandcenter-webapi-platform`: the same image, `Hosting__Role=platform`, its own snapshot volume, no `CommandCenter` database
`WEBAPI_ROLE`	`environment`	the environment's WebApi stops serving and writing the platform's pages
`PLATFORM_API_URL`	`http://commandcenter-webapi-platform/`	the gateway's `cluster-platform`
`OPA_ACCESS_BUNDLE_URL`	`http://commandcenter-webapi-platform`	OPA downloads who may do what from the platform host. Left on the environment's WebApi it gets 404, and decides with what it persisted or the release defaults.

Identities. The platform host needs: - App Configuration Data Owner; - Key Vault Secrets Officer (signing keys); - Key Vault Certificates Officer (TLS certificates, docs/certificates.md); - Storage Blob Data Contributor on config-status.

The environment hosts then need only App Configuration Data Reader, Key Vault Secrets User and Storage Blob Data Contributor on config-status. That is the point of the split: an environment host can no longer change the platform. The dev Azure stack (docs/dev-azure.md) has one service principal per developer for every container, so it does not exercise this split.

The agent registry moves with the platform. In compose both services mount the volume platform-data with the same Platform__Database__Path; only the one serving the platform opens it. The environment WebApi then reads the registry from the platform host at PLATFORM_API_URL, with AGENT_REGISTRY_TOKEN (made by scripts/dev-bootstrap.ps1). Without the token every page that needs the agents answers 503 on the environment host.

Order. 1. Grant the platform host's identity its roles. 2. Start the platform host (the profile) and check its /health/ready. 3. Point the gateway and OPA at it. 4. Give the platform host and the environment hosts the same Agents:Registry:Token, and the environment hosts Agents:Registry:Url. 5. Set WEBAPI_ROLE=environment on the environment hosts, and check their /health/ready (the agent registry is ready once the first directory arrived).

Going back is the reverse: WEBAPI_ROLE unset makes the environment's WebApi combined again.

Setting up a host#

Setting	Example
`Configuration__AppConfig__Endpoint`	`https://cfg-cc-prod.azconfig.io`	the store; the managed identity signs in
`Configuration__Environment`	`prod`	the environment label (defaults to the host's environment name)
`Configuration__KeyVault__Uri`	`https://kv-cc-prod.vault.azure.net/`	the only vault references resolve against
`Configuration__Status__BlobServiceUri`	`https://stccprod.blob.core.windows.net/`	where status is written
`Configuration__Identity__ManagedIdentityClientId`	a client id	pick a user-assigned identity, when the VM has several
`Configuration__Identity__TenantId` / `ClientId` / `ClientSecret` or `CertificatePath`		an alternate identity instead of the managed one

The identity needs App Configuration Data Reader on the store, Key Vault Secrets User on the vault, and Storage Blob Data Contributor on the config-status container. The WebApi that serves the platform (the platform host, or a combined WebApi) also needs App Configuration Data Owner and Key Vault Secrets Officer (see Running a platform host).

When App Configuration is down#

Every host keeps a last-known-good snapshot of its labels (Configuration__Snapshot__Path, on a volume; compose sets /var/lib/commandcenter/config-snapshot.json for the gateway, main API and WebApi). It is written after every load and every applied revision, and holds the key-values as the store has them — secrets stay Key Vault references, so no secret is ever on disk.

flowchart TD
    Start([host starts]) --> Probe{App Configuration answers?}
    Probe -- yes --> Live[load from the store, write the snapshot]
    Probe -- no --> Snap{snapshot younger than MaxAge?}
    Snap -- yes --> Cached[start from the snapshot<br/>resolve references from Key Vault<br/>status: Cached]
    Snap -- no --> Optional{Optional?}
    Optional -- yes --> Local[start on local settings<br/>status: Unreachable]
    Optional -- no --> Fail[fail; Docker retries]
    Cached --> Wait[ask the store every 30 s]
    Local --> Wait
    Wait -- answers --> Restart[stop after a random pause<br/>Docker restarts it on the store]

Hold "Alt" / "Option" to enable pan & zoom

A snapshot older than Configuration__Snapshot__MaxAge (7 days) is not used: the host fails rather than run on settings that old.
A host that started from the snapshot (or without the store) stops itself once the store answers again, after a random pause so a service's instances do not all restart at once, and Docker (restart: unless-stopped) starts it on the store's settings — nobody has to log in. Configuration__Snapshot__RestartWhenStoreReturns=false turns that off (the development stack does, because dotnet watch does not exit when the app stops).
Without a snapshot, a host configured for App Configuration that cannot reach it fails after Configuration:AppConfig:StartupTimeout (30 s) and Docker retries it. Configuration__AppConfig__Optional=true lets it start on its local settings instead (development).
Never store a secret as a plain value in App Configuration: it would be in the snapshot. The main API's and WebApi's former ConnectionStrings:AppConfig still works, and WebApi still reads its former label CommandCenter.WebApi.

Locally#

compose.dev.yaml runs the App Configuration emulator (appconfig, web UI on http://localhost:8483) and points the gateway, the main API and WebApi at it with environment dev; status goes to Azurite. To try a change: create a key with label dev, then bump Settings:Sentinel (label dev) — the service picks it up within a few seconds.

In Azure mode (docs/dev-azure.md) the same hosts read your own App Configuration store instead, a Developer-tier store per developer, so the labels (platform, dev, dev/{service}) and the environment name stay the same. compose.azure.yaml sets only the bootstrap keys; scripts/dev-azure/connect.ps1 puts the rest in the store at label dev. The refresh interval there is 2 minutes.

Shared configuration#

The settings pages on /config#

Changing a setting#

What each host reads#

When a change applies#

What applies without a restart#

Host status#

The Hosts panel (/config/hosts)#

WebApi's roles (Hosting:Role)#

Running a platform host#

Setting up a host#

When App Configuration is down#

Locally#

The settings pages on `/config`#

The Hosts panel (`/config/hosts`)#

WebApi's roles (`Hosting:Role`)#