Skip to content

Agents: registration, transport and authentication#

Every server that hosts BenefitManager runs a Benefits agent. CommandCenter (WebApi) uses it to see what is installed, to deploy and remove versions, and to manage services, IIS and certificates. This page is how the two find each other, how they talk, and how they trust each other without anyone having to touch the servers again once an agent is installed.

  • Two channels. Azure Service Bus is the preferred one: agents publish a heartbeat and receive commands on topics, with nothing listening on the server. Direct HTTP is the other, for servers where that is simpler — QA, UAT, development.
  • One registry, the platform's. /config/agents lists every agent WebApi may address, whichever environment the page is open in: the agents and IIS hosts are shared by the environments. Service Bus agents register themselves with their first heartbeat; HTTP agents are added by address. Both are pending until an operator accepts them.
  • Servers are the environment's. Which agent manages a server is set per environment on /config/servers (TblServerAgent), from the accepted agents. One agent can manage several of an environment's servers.
  • No shared secret. Over HTTP, agents trust CommandCenter's client certificate by its issuer and subject, so it can be renewed without changing the agents. The old API key is deprecated and off.

Decisions behind this: ADR 0007 (registry and transport), #309 (agents report themselves; health), #310 (mTLS), #311 (revocation, open).

Context#

flowchart LR
    operator(Operator)

    subgraph azure [Azure]
        direction TB
        bff((BFF<br/>YARP gateway))
        webapi((CommandCenter<br/>WebApi))
        pdb[|borders:tb|Platform db · SQLite<br/>TblAgent · TblAgentHeartbeat]
        db[|borders:tb|Environment db<br/>TblServer · TblServerAgent]
        kv[(Key Vault<br/>WebApi client certificate)]
        sb{{Service Bus<br/>agent.state · agent.commands · agent.replies}}

        bff -- "/api/registry · /api/bmo/*" --> webapi
        webapi <--> pdb
        webapi <--> db
        webapi -. "managed identity:<br/>latest certificate" .-> kv
        webapi <-- "commands / replies" --> sb
        sb -- "heartbeats" --> webapi
    end

    subgraph onprem [Customer servers]
        direction TB
        subgraph server [Windows server]
            agent((Benefits agent<br/>:5010))
            iis[IIS sites · BenefitManager services]
            store[|borders:tb|Machine certificate store<br/>BCS Root + Intermediates<br/>agent server certificate]
            agent --> iis
            agent -.-> store
        end
    end

    subgraph pki [BCS PKI]
        direction TB
        root[Root CA<br/>self-signed, installed everywhere]
        clientCa[Client Issuing CA]
        serverCa[Server Issuing CA]
        root --> clientCa
        root --> serverCa
    end

    operator -- "browser" --> bff
    webapi == "HTTPS + mTLS<br/>(HTTP agents)" ==> agent
    agent <-- "heartbeat · commands<br/>(Service Bus agents)" --> sb
    clientCa -. "issues WebApi's<br/>client certificate" .-> kv
    serverCa -. "issues the agent's<br/>server certificate" .-> store

%% Element type definitions

    classDef boundary fill:none
    azure:::boundary
    onprem:::boundary
    classDef trust stroke-dasharray: 5 5
    pki:::trust
Hold "Alt" / "Option" to enable pan & zoom

Solid arrows are requests; dotted arrows are where certificates come from. WebApi's client certificate lives in Key Vault (or the Windows store, or a file on Linux) and is re-read hourly; the agent's server certificate lives in the machine's store and is chosen by subject at every handshake. Neither side pins a thumbprint, so a renewal from the same CA needs no change on the other side.

Where the registry lives#

The agents, their heartbeats and what they reported are in the platform database: a SQLite file (Platform:Database:Path, in compose the volume platform-data at /var/lib/commandcenter-platform/platform.db) that only the WebApi serving the platform opens (Hosting:Role combined or platform). That host migrates it when it starts; nothing else does, and GET /api/maintenance/migrate does not touch it. Exactly one such instance per deployment: a second one would keep a registry of its own. The environment database keeps the servers and, in TblServerAgent, which agent manages each, by the agent's name.

flowchart LR
    subgraph platform [Host serving the platform]
        reg["/api/registry/agents"]
        rec["heartbeat consumer"]
        ia["/internal/agents/*<br/>(Agents:Registry:Token)"]
        pdb[(platform.db<br/>TblAgent · TblAgentHeartbeat)]
        reg --> pdb
        rec --> pdb
        ia --> pdb
    end
    subgraph env [Environment dev / uat / prod]
        srv["/api/registry/servers<br/>PUT …/{key}/agent"]
        job["server-availability job"]
        edb[(TblServer · TblServerAgent)]
        srv --> edb
        job --> edb
    end
    srv -- "GET /internal/agents/reports<br/>(live, per request)" --> ia
    job -- "GET /internal/agents/reports" --> ia
    cmd["operator commands<br/>(Restart, /api/agents/**)"] -- "directory snapshot<br/>GET /internal/agents/directory every 5 s" --> ia
Hold "Alt" / "Option" to enable pan & zoom

A combined host reads all of it in-process. An environment host (Hosting:Role=environment) never opens the file; it reads the platform host's /internal/agents/* at Agents:Registry:Url, with the bearer token Agents:Registry:Token (the same value on both; compose: AGENT_REGISTRY_TOKEN). These routes are not reachable through the gateway and not in the OpenAPI document; a platform host without a token answers 503, a wrong token 401.

  • Commands (directory and heartbeat times) use a snapshot reloaded every 5 seconds (ETag, 304 when unchanged). When the platform host is away the last snapshot stays: HTTP agents keep working, and a Service Bus agent is refused once its last known heartbeat is older than MaxAgentStateAge (15 minutes) — the reason then reads "stale heartbeat". The host's /health/ready reports the directory as degraded after 30 seconds without an answer, and as not ready before the first one.
  • Pages and the availability job (servers, fleet state, Restart) ask /internal/agents/reports live, never from a copy. When it cannot be read they answer 503 agent-registry-unavailable, and the availability job records the run as failed ("Skipped, no server changed"): a server is never set offline on an old or missing registry.
  • Heartbeats are judged by the host that records them. The reports say whether the platform host records heartbeats (it has Agents:ServiceBus) and with which MaxAgentStateAge. Where it does not, a Service Bus agent's missing heartbeat means "cannot be judged", never offline — even when the environment host itself has Service Bus for sending commands.
  • Service Bus is split by role. The heartbeat consumer (and self-registration) runs only where the registry is; the command transport runs wherever an environment is served. WebApi's client certificate is refreshed on every role, because every role calls agents over HTTP.

Data flow#

Before any of this: the agent prepares its IIS sites#

When the agent service starts it creates or repairs the IIS site for every configured environment — binding, root application pool, physical path, and read access for the pool identities — so a deployment never fails on a missing Acceptatie or Productie site. An environment it could not prepare is reported as Degraded health (visible in the health the flows below carry and on /config/agents) and deployments to that environment are refused with the reason; the other environments are unaffected. Details, settings and log ids: the agent setup guide.

Adding an HTTP agent#

The operator types only the address. WebApi completes it, asks the agent who it is, and records what it reports.

sequenceDiagram
    autonumber
    actor Op as Operator
    participant SPA as /config/agents
    participant API as WebApi
    participant Agent as Agent (tas-bm02)
    participant DB as Platform db (TblAgent)

    Op->>SPA: address "tas-bm02" (name left empty)
    SPA->>API: POST /api/registry/agents {name: null, httpBaseUrl}
    API->>API: complete to https://tas-bm02:5010
    API->>Agent: GET /health (anonymous)
    Agent-->>API: 200 Healthy / 200 Degraded / 503 Unhealthy
    Note over API,Agent: any stated health is reachable
    API->>Agent: GET /api/installations (mTLS)
    Agent-->>API: 200 [installations]
    API->>Agent: GET /api/agent/info (mTLS)
    Agent-->>API: {name, hostname, health, bindings}
    API->>DB: insert pending agent: reported name, host, bindings, health, verified
    API-->>SPA: agent (pending)
    SPA-->>Op: name, "HTTP verified", health badge, its IIS sites
Hold "Alt" / "Option" to enable pan & zoom

If the agent cannot report its name — an older agent, or one that is not reachable yet — WebApi asks for it instead of guessing. Verify HTTP runs steps 4–9 again and refreshes what is stored.

The mTLS handshake#

sequenceDiagram
    participant API as WebApi ("Agent" client)
    participant Agent as Agent (Kestrel)
    participant Auth as Agent certificate policy

    API->>Agent: TLS ClientHello
    Agent-->>API: server certificate (newest valid for ServerCertificate:Subject)
    API->>API: validate it against the trusted BCS chain in the image
    API->>Agent: client certificate + Client Issuing CA (never the root)
    Agent->>Auth: validated chain
    Auth->>Auth: chained, not self-signed?
    Auth->>Auth: TrustedIssuers contains "Client Issuing CA"?
    Auth->>Auth: subject in AllowedSubjects?
    alt all three
        Auth-->>Agent: authenticated
        Agent-->>API: 200
    else otherwise (or nothing configured)
        Auth-->>Agent: rejected, reason logged
        Agent-->>API: 401 / 403
    end
Hold "Alt" / "Option" to enable pan & zoom

TrustedIssuers names the client intermediate, not the root: the same root also signs the server intermediate, and a server certificate must not be able to authenticate as a client. With no AllowedSubjects (and no thumbprints) configured, the agent accepts no certificate.

Service Bus agents#

sequenceDiagram
    participant Agent
    participant SB as Service Bus
    participant API as WebApi
    participant DB as Platform db (TblAgentHeartbeat)

    loop every monitoring interval (5 min)
        Agent->>SB: agent.state.v1 (name, host, installations, IIS bindings)
        SB->>API: subscription commandcenter-webapi
        API->>DB: upsert by name (liveness = enqueue time)
    end
    Note over API: unknown name → a pending agent on /config/agents
    API->>SB: agent.commands.v1 (only for an agent with a fresh heartbeat)
    SB->>Agent: command
    Agent->>SB: agent.replies.v1
    SB->>API: reply
Hold "Alt" / "Option" to enable pan & zoom

GET /api/bmo/fleet/state is served from these heartbeats for Service Bus agents (asked live for HTTP agents), accepted agents only; a heartbeat older than 15 minutes means offline.

Certificate rollover#

flowchart LR
    renew["Key Vault / auto-enrollment<br/>renews a certificate<br/>(same issuer, same subject)"]
    renew --> kind{which one?}
    kind -- "WebApi's client certificate" --> refresh["WebApi re-reads it hourly;<br/>new 'Agent' handlers take it<br/>within 2 minutes"]
    kind -- "agent's server certificate" --> select["agent picks the newest valid one<br/>by subject at the next handshake<br/>(re-read hourly)"]
    refresh --> accepted["agents accept it:<br/>same issuer + subject"]
    select --> trusted["WebApi trusts it:<br/>chains to the BCS root in its image"]
    accepted --> done([nobody touches a server])
    trusted --> done
Hold "Alt" / "Option" to enable pan & zoom

What health decides#

flowchart TD
    health{agent health} -->|Healthy| ok["/health 200<br/>everything allowed"]
    health -->|"Degraded<br/>(< 10% free memory or disk)"| degraded["/health 200<br/>everything allowed, badge shown"]
    health -->|"Unhealthy<br/>(< 5% free, measured)"| unhealthy["/health 503<br/>deployments refused<br/>status, services, IIS, certificates,<br/>uninstall still work"]
    health -->|"check could not run<br/>(e.g. WMI)"| unknown["logged<br/>deployments not blocked"]
Hold "Alt" / "Option" to enable pan & zoom

An Unhealthy agent refuses to deploy BenefitManager versions — over HTTP (503 with the reason) and over Service Bus (a failed reply) — and nothing else, because the other commands are how an operator frees the machine.

A refusal is never a success. Over Service Bus every command waits for the agent's reply, and a reply that says it failed — an error, a refusal, or 501 from an agent too old to know the command — answers 503 with the agent's reason on /api/bmo/agents/**, for the writes (remove a version, start/stop/restart a service, recycle/start/stop an application pool, upload or delete a certificate) as for the reads. Before #358 the writes reported such a reply as done. Restart is the one command whose refusal is an answer of its own (409, see the Configuration tab below).

User journey#

journey
    title An operator brings a new server under CommandCenter
    section Once per server
      Install agent: 3: Operations
      Server cert: 3: Operations
      Allow WebApi: 3: Operations
    section Register
      Type address: 5: Operator
      Agent names itself: 5: Operator
      Verified + health: 4: Operator
      Accept: 5: Operator
    section Each environment
      Link its servers: 4: Operator
    section Operate
      Fleet + deploys: 5: Operator
      Maintenance, IIS: 4: Operator
      Unhealthy, no deploy: 3: Operator
      Free disk, deploy: 4: Operator
    section Years later
      Certs renew alone: 5: Operator, Operations
Hold "Alt" / "Option" to enable pan & zoom

Once per server, operations installs the agent, gives it a server certificate by subject (the BCS root and intermediates are already on the machine) and allows WebApi's subject and the client CA in ClientAuthentication. From then on the operator works only on /config/agents: types the address, sees the agent name itself and its IIS sites, verifies and accepts. Then, in each environment the server serves, /config/servers links the server rows to the agent (Edit → Agent). Day to day the server shows up on the fleet dashboard and the deployment page; an Unhealthy agent refuses deployments with the reason until disk or memory is freed. Certificates renew without anyone doing anything.

Central settings and restart (Configuration tab)#

The Configuration tab of /config/agents manages the agents' own settings in App Configuration, and restarts an agent. Nobody signs in to a server for it. The agent side is in the agent guide.

flowchart LR
    Operator -- "Configuration tab" --> SPA
    SPA -- "PUT /api/registry/agents/settings<br/>(agents.config.write)" --> WebApi
    SPA -- "POST /api/registry/agents/{key}/restart<br/>(hosts.restart)" --> WebApi
    WebApi -- "platform/agent<br/>platform/agent/{name}<br/>+ Settings:PlatformSentinel" --> AC["App Configuration"]
    AC -- "reload on the sentinel" --> Agent
    WebApi -- "Restart (Service Bus or HTTP)" --> Agent
    Agent -- "heartbeat: configuration state" --> SB["Service Bus"] --> WebApi
    WebApi -- "TblAgentHeartbeat.ReportedConfiguration" --> DB[(platform db)]
Hold "Alt" / "Option" to enable pan & zoom

What the tab shows per agent, as the agent last reported it (with every heartbeat over Service Bus, or when an HTTP agent was added or verified; key names only): - the source: AppConfiguration, LocalOnly, or Unreachable; - the revision (the platform sentinel it applied) and when it applied it; - keys waiting for a restart; - keys overridden on the server; - Key Vault references that did not resolve; - the last error; - Restart, or the reason the agent cannot restart on request. The usual reason is that service recovery is not configured: run Install.ps1 on that server.

The settings are the catalogue in bmo/src/Benefits.Configuration/AgentSettingsCatalogue.cs, the same one the agent uses to tell a live change from a restart-bound one:

Setting Applies
Agent:MonitoringInterval live
Agent:ProtectedAppPools live
ClientAuthentication:AllowedSubjects, TrustedIssuers live
Serilog:MinimumLevel:Default, Serilog:MinimumLevel:Override:{category} live
ClientAuthentication:RevocationMode restart
ServerCertificate:Subject restart
Agent:ServiceBus:Namespace, Agent:ServiceBus:SnapshotContainerUrl restart
PackageStore:AccountUri restart
Agent:PrepareEnvironmentsOnStartup restart
  • Where a setting goes. Choose Every agent (label platform/agent) or one agent (label platform/agent/{name}, the name in lower case), which wins. For one agent, the page shows what every agent has next to each field.
  • Lists are stored as one JSON array value without a JSON content type, so the agent replaces its local list rather than merging it by index.
  • Validation runs server-side, per kind: a duration between 10 s and a day, true/false, one of the choices, a JSON array of strings. A key outside the catalogue is refused; one that is already at the label can only be removed.
  • The keys an agent is reached or deployed by (Agent:ServiceBus:Namespace, ServerCertificate:Subject, PackageStore:AccountUri) are per agent only: a save for every agent is refused (400). A wrong value would otherwise cut off the whole fleet at its next restart, where fixing it centrally needs yet another restart. One stored for every agent before can still be removed. They are also checked for their form: the namespace must be a fully qualified host name, and the package store and Agent:ServiceBus:SnapshotContainerUrl must be https addresses without a query (no SAS token).
  • A save is conditional on the version read: a changed label answers 409 and nothing is written. It writes only the keys that changed, bumps Settings:PlatformSentinel, and writes audit event 3197. If the save fails partway (someone else changed a key, or the store stopped answering), the keys written before the failure stay stored. The sentinel is bumped for them too, so the agents load them, and the page says the save was partial.
  • HTTP agents have no heartbeat. What they report is refreshed by Check (the HTTP verification), which the tab offers next to them, also after a Restart.
  • Restart (hosts.restart) goes to an accepted agent over Service Bus or HTTP, as it is reached. 202: the agent stops and Windows service recovery starts it again. 409 carries the agent's reason: no recovery, not a Windows service, a deployment running or queued, or an agent too old to know the command. 503 when the agent cannot be reached. Audit events 3198 (restarted) and 3199 (refused).
journey
    title An operator shortens every agent's heartbeat and moves one to a new namespace
    section Every agent
      Open Configuration tab: 5: Operator
      Set MonitoringInterval (live): 5: Operator
      Save; agents reload within a minute: 5: Operator
    section One agent
      Choose the agent: 5: Operator
      Set ServiceBus Namespace (restart): 4: Operator
      Save; key shows as waiting for a restart: 4: Operator
      Restart; agent answers 202: 5: Operator
      Next heartbeat: new start, nothing waiting: 5: Operator
Hold "Alt" / "Option" to enable pan & zoom

Configuration#

WebApi

Setting Meaning
Agents:ClientCertificate:Source KeyVault (with KeyVaultUri, CertificateName; latest version, managed identity), Store (with Subject; newest valid with a private key) or File (with FilePath, Password; re-read when it changes). Empty: no certificate.
Agents:ClientCertificate:RefreshInterval How often it is re-read. Default 01:00:00.
Agents:ServiceBus:Namespace / ConnectionString The Service Bus channel; off while both are empty. Set or changed while WebApi runs, it applies within about 15 seconds, without a restart.
BenefitsAgent:ApiKey Deprecated. Sent as X-Api-Key only while set.

Agent (details in the agent guide)

Setting Meaning
ServerCertificate:Subject The agent's HTTPS certificate, chosen by subject from the store.
ClientAuthentication:AllowedSubjects WebApi's certificate subject. Required — empty accepts nothing.
ClientAuthentication:TrustedIssuers The client issuing CA (not the root).
ClientAuthentication:RevocationMode NoCheck for the BCS PKI until it publishes revocation (#311).
Authentication:ApiKey Deprecated; the scheme exists only while set.
Agent:Name, Agent:Port Reported name (default the machine name) and port (default 5010).

Rolling it out#

  1. Apply the migrations (GET /api/maintenance/migrate): AddAgentRegistry, AddAgentHeartbeats, AddHeartbeatInstallations, AddAgentReportedInfo, AddAgentConfigState.
  2. Issue WebApi's client certificate from the client CA with a fixed subject; set Agents__ClientCertificate__*.
  3. On each server, once: the agent's server certificate by subject, ClientAuthentication, and an agent release that has /api/agent/info, the health gate and the fail-closed policy.
  4. Add or verify the HTTP agents on /config/agents with https:// addresses; accept them.
  5. Remove the API key everywhere; the key once committed to the agent's settings (123-abcd) is public and must be replaced wherever it was used.

Moving the registry to the platform database (#363)#

The registry starts empty in the platform database; nothing is copied from the old tables.

  1. Deploy with every WebApi on Hosting:Role combined (the default) or platform, and one instance serving the platform. Give it a volume for Platform:Database:Path that survives a re-deployment: the registry is in it.
  2. Apply MoveAgentRegistryToPlatform in each environment (GET /api/maintenance/migrate). It drops dbo.TblAgent and dbo.TblAgentHeartbeat and creates TblServerAgent. There is no way back to the old tables with their data: note the accepted agents first if you want a list.
  3. Service Bus agents register again with their next heartbeat (within 5 minutes); add the HTTP agents again by address. Accept them on /config/agents.
  4. In each environment, link the servers to their agents on /config/servers. Until a server is linked, the availability job leaves it alone.