Skip to content

Shared configuration#

CommandCenter's services read their settings from Azure App Configuration, with secrets in Azure Key Vault. A change made there reaches every instance of every service within about half a minute — without anyone signing in to a server or editing a file. The /config pages will be where operators make those changes; until they are, the store is edited in the portal or with the CLI.

What each host reads#

flowchart LR
    subgraph local ["On the VM (set once at deployment)"]
        Boot["Configuration:*<br/>endpoint, identity, environment"]
        Env["environment variables<br/>(break-glass, reported)"]
    end
    subgraph azure ["Azure"]
        AppConfig["App Configuration<br/>labels: platform, {env}, {env}/{service}"]
        KeyVault["Key Vault<br/>secrets"]
        Status["Blob container config-status"]
    end
    Host["Service instance"]

    Boot --> Host
    AppConfig -- "settings, refreshed on the sentinel" --> Host
    KeyVault -- "Key Vault references" --> Host
    Env -- "wins over App Configuration" --> Host
    Host -- "what it loaded, every minute" --> Status
Hold "Alt" / "Option" to enable pan & zoom
  • Bootstrap, on the VM (Configuration:*): only what a host needs to find the store — its endpoint, the identity to use, the environment name. Nothing else lives in .env or appsettings.json on a server.
  • Labels, later wins: no label → platform → {env} → {env}/{service} (a host serving every environment, like the gateway: no label → platform → platform/{service}). Keys keep the paths the code reads (LogAnalytics:WorkspaceId); the label decides which hosts get a value.
  • Order: appsettings.json → App Configuration → environment variables → command line. An environment variable on a VM still wins — the per-server break-glass — and is reported as a shadowed key, so a change in the UI that seems to have no effect can be explained.
  • Secrets are Key Vault references in App Configuration, resolved with the host's identity, from the configured vault only. A reference that cannot be read becomes an empty value and is reported; it never stops a host.

When a change applies#

sequenceDiagram
    participant Operator
    participant AppConfig as App Configuration
    participant Host as Every instance
    participant Status as config-status
    Operator->>AppConfig: change settings (one or several)
    Operator->>AppConfig: bump the sentinel
    loop every 30 s
        Host->>AppConfig: did a sentinel move?
    end
    AppConfig-->>Host: yes: reload all settings at once
    Host->>Status: revision 0/7, restart pending: ConnectionStrings:…
Hold "Alt" / "Option" to enable pan & zoom
  • Nothing reloads until a sentinel moves: Settings:Sentinel (label {env}) for an environment, Settings:PlatformSentinel (label platform) for the platform. A batch of changes therefore applies together.
  • A host reloads all its settings when a sentinel moves. Whether a setting takes effect at once depends on the code that reads it; connection strings, sign-in (Authentication, AzureAd) and ports (Kestrel, Urls) do not, and the host reports them as restart pending.
  • The revision a host reports is its sentinels' values (platform/environment, e.g. 3/7).

Host status#

Every instance writes {env}/{service}/{instance}.json (or platform/{service}/…) to the config-status blob container every minute and right after a change:

{ "service": "webapi", "environment": "prod", "instance": "vm-cc-01", "source": "AppConfiguration",
  "revision": "3/7", "startupRevision": "3/5", "restartPendingKeys": ["ConnectionStrings:CommandCenter"],
  "shadowedKeys": [], "unresolvedSecrets": [] }

Only key names — never values. The Hosts panel on /config reads these; until then, the container shows which instance runs which revision and which needs a restart.

Setting up a host#

Setting Example
Configuration__AppConfig__Endpoint https://cfg-cc-prod.azconfig.io the store; the managed identity signs in
Configuration__Environment prod the environment label (defaults to the host's environment name)
Configuration__KeyVault__Uri https://kv-cc-prod.vault.azure.net/ the only vault references resolve against
Configuration__Status__BlobServiceUri https://stccprod.blob.core.windows.net/ where status is written
Configuration__Identity__ManagedIdentityClientId a client id pick a user-assigned identity, when the VM has several
Configuration__Identity__TenantId / ClientId / ClientSecret or CertificatePath an alternate identity instead of the managed one

The identity needs App Configuration Data Reader on the store, Key Vault Secrets User on the vault, and Storage Blob Data Contributor on the config-status container.

When App Configuration is down#

Every host keeps a last-known-good snapshot of its labels (Configuration__Snapshot__Path, on a volume; compose sets /var/lib/commandcenter/config-snapshot.json for the gateway, main API and WebApi). It is written after every load and every applied revision, and holds the key-values as the store has them — secrets stay Key Vault references, so no secret is ever on disk.

flowchart TD
    Start([host starts]) --> Probe{App Configuration answers?}
    Probe -- yes --> Live[load from the store, write the snapshot]
    Probe -- no --> Snap{snapshot younger than MaxAge?}
    Snap -- yes --> Cached[start from the snapshot<br/>resolve references from Key Vault<br/>status: Cached]
    Snap -- no --> Optional{Optional?}
    Optional -- yes --> Local[start on local settings<br/>status: Unreachable]
    Optional -- no --> Fail[fail; Docker retries]
    Cached --> Wait[ask the store every 30 s]
    Local --> Wait
    Wait -- answers --> Restart[stop after a random pause<br/>Docker restarts it on the store]
Hold "Alt" / "Option" to enable pan & zoom
  • A snapshot older than Configuration__Snapshot__MaxAge (7 days) is not used: the host fails rather than run on settings that old.
  • A host that started from the snapshot (or without the store) stops itself once the store answers again, after a random pause so a service's instances do not all restart at once, and Docker (restart: unless-stopped) starts it on the store's settings — nobody has to log in. Configuration__Snapshot__RestartWhenStoreReturns=false turns that off (the development stack does, because dotnet watch does not exit when the app stops).
  • Without a snapshot, a host configured for App Configuration that cannot reach it fails after Configuration:AppConfig:StartupTimeout (30 s) and Docker retries it. Configuration__AppConfig__Optional=true lets it start on its local settings instead (development).
  • Never store a secret as a plain value in App Configuration: it would be in the snapshot. The main API's and WebApi's former ConnectionStrings:AppConfig still works, and WebApi still reads its former label CommandCenter.WebApi.

Locally#

compose.dev.yaml runs the App Configuration emulator (appconfig, web UI on http://localhost:8483) and points the gateway, the main API and WebApi at it with environment dev; status goes to Azurite. To try a change: create a key with label dev, then bump Settings:Sentinel (label dev) — the service picks it up within a few seconds.