Skip to content

Agents: registration, transport and authentication#

Every server that hosts BenefitManager runs a Benefits agent. CommandCenter (WebApi) uses it to see what is installed, to deploy and remove versions, and to manage services, IIS and certificates. This page is how the two find each other, how they talk, and how they trust each other without anyone having to touch the servers again once an agent is installed.

  • Two channels. Azure Service Bus is the preferred one: agents publish a heartbeat and receive commands on topics, with nothing listening on the server. Direct HTTP is the other, for servers where that is simpler — QA, UAT, development.
  • One registry. /config/agents lists every agent WebApi may address. Service Bus agents register themselves with their first heartbeat; HTTP agents are added by address. Both are pending until an operator accepts them, which links them to their server.
  • No shared secret. Over HTTP, agents trust CommandCenter's client certificate by its issuer and subject, so it can be renewed without changing the agents. The old API key is deprecated and off.

Decisions behind this: ADR 0007 (registry and transport), #309 (agents report themselves; health), #310 (mTLS), #311 (revocation, open).

Context#

flowchart LR
    operator(Operator)

    subgraph azure [Azure]
        direction TB
        bff((BFF<br/>YARP gateway))
        webapi((CommandCenter<br/>WebApi))
        db[|borders:tb|CommandCenter db<br/>TblAgent · TblAgentHeartbeat]
        kv[(Key Vault<br/>WebApi client certificate)]
        sb{{Service Bus<br/>agent.state · agent.commands · agent.replies}}

        bff -- "/api/registry · /api/bmo/*" --> webapi
        webapi <--> db
        webapi -. "managed identity:<br/>latest certificate" .-> kv
        webapi <-- "commands / replies" --> sb
        sb -- "heartbeats" --> webapi
    end

    subgraph onprem [Customer servers]
        direction TB
        subgraph server [Windows server]
            agent((Benefits agent<br/>:5010))
            iis[IIS sites · BenefitManager services]
            store[|borders:tb|Machine certificate store<br/>BCS Root + Intermediates<br/>agent server certificate]
            agent --> iis
            agent -.-> store
        end
    end

    subgraph pki [BCS PKI]
        direction TB
        root[Root CA<br/>self-signed, installed everywhere]
        clientCa[Client Issuing CA]
        serverCa[Server Issuing CA]
        root --> clientCa
        root --> serverCa
    end

    operator -- "browser" --> bff
    webapi == "HTTPS + mTLS<br/>(HTTP agents)" ==> agent
    agent <-- "heartbeat · commands<br/>(Service Bus agents)" --> sb
    clientCa -. "issues WebApi's<br/>client certificate" .-> kv
    serverCa -. "issues the agent's<br/>server certificate" .-> store

%% Element type definitions

    classDef boundary fill:none
    azure:::boundary
    onprem:::boundary
    classDef trust stroke-dasharray: 5 5
    pki:::trust
Hold "Alt" / "Option" to enable pan & zoom

Solid arrows are requests; dotted arrows are where certificates come from. WebApi's client certificate lives in Key Vault (or the Windows store, or a file on Linux) and is re-read hourly; the agent's server certificate lives in the machine's store and is chosen by subject at every handshake. Neither side pins a thumbprint, so a renewal from the same CA needs no change on the other side.

Data flow#

Adding an HTTP agent#

The operator types only the address. WebApi completes it, asks the agent who it is, and records what it reports.

sequenceDiagram
    autonumber
    actor Op as Operator
    participant SPA as /config/agents
    participant API as WebApi
    participant Agent as Agent (tas-bm02)
    participant DB as TblAgent

    Op->>SPA: address "tas-bm02" (name left empty)
    SPA->>API: POST /api/registry/agents {name: null, httpBaseUrl}
    API->>API: complete to https://tas-bm02:5010
    API->>Agent: GET /health (anonymous)
    Agent-->>API: 200 Healthy / 200 Degraded / 503 Unhealthy
    Note over API,Agent: any stated health is reachable
    API->>Agent: GET /api/installations (mTLS)
    Agent-->>API: 200 [installations]
    API->>Agent: GET /api/agent/info (mTLS)
    Agent-->>API: {name, hostname, health, bindings}
    API->>DB: insert pending agent: reported name, host, bindings, health, verified
    API-->>SPA: agent (pending)
    SPA-->>Op: name, "HTTP verified", health badge, server URLs to choose from
Hold "Alt" / "Option" to enable pan & zoom

If the agent cannot report its name — an older agent, or one that is not reachable yet — WebApi asks for it instead of guessing. Verify HTTP runs steps 4–9 again and refreshes what is stored.

The mTLS handshake#

sequenceDiagram
    participant API as WebApi ("Agent" client)
    participant Agent as Agent (Kestrel)
    participant Auth as Agent certificate policy

    API->>Agent: TLS ClientHello
    Agent-->>API: server certificate (newest valid for ServerCertificate:Subject)
    API->>API: validate it against the trusted BCS chain in the image
    API->>Agent: client certificate + Client Issuing CA (never the root)
    Agent->>Auth: validated chain
    Auth->>Auth: chained, not self-signed?
    Auth->>Auth: TrustedIssuers contains "Client Issuing CA"?
    Auth->>Auth: subject in AllowedSubjects?
    alt all three
        Auth-->>Agent: authenticated
        Agent-->>API: 200
    else otherwise (or nothing configured)
        Auth-->>Agent: rejected, reason logged
        Agent-->>API: 401 / 403
    end
Hold "Alt" / "Option" to enable pan & zoom

TrustedIssuers names the client intermediate, not the root: the same root also signs the server intermediate, and a server certificate must not be able to authenticate as a client. With no AllowedSubjects (and no thumbprints) configured, the agent accepts no certificate.

Service Bus agents#

sequenceDiagram
    participant Agent
    participant SB as Service Bus
    participant API as WebApi
    participant DB as TblAgentHeartbeat

    loop every monitoring interval (5 min)
        Agent->>SB: agent.state.v1 (name, host, installations, IIS bindings)
        SB->>API: subscription commandcenter-webapi
        API->>DB: upsert by name (liveness = enqueue time)
    end
    Note over API: unknown name → a pending agent on /config/agents
    API->>SB: agent.commands.v1 (only for an agent with a fresh heartbeat)
    SB->>Agent: command
    Agent->>SB: agent.replies.v1
    SB->>API: reply
Hold "Alt" / "Option" to enable pan & zoom

GET /api/bmo/fleet/state is served from this table for Service Bus agents (asked live for HTTP agents), accepted agents only; a heartbeat older than 15 minutes means offline.

Certificate rollover#

flowchart LR
    renew["Key Vault / auto-enrollment<br/>renews a certificate<br/>(same issuer, same subject)"]
    renew --> kind{which one?}
    kind -- "WebApi's client certificate" --> refresh["WebApi re-reads it hourly;<br/>new 'Agent' handlers take it<br/>within 2 minutes"]
    kind -- "agent's server certificate" --> select["agent picks the newest valid one<br/>by subject at the next handshake<br/>(re-read hourly)"]
    refresh --> accepted["agents accept it:<br/>same issuer + subject"]
    select --> trusted["WebApi trusts it:<br/>chains to the BCS root in its image"]
    accepted --> done([nobody touches a server])
    trusted --> done
Hold "Alt" / "Option" to enable pan & zoom

What health decides#

flowchart TD
    health{agent health} -->|Healthy| ok["/health 200<br/>everything allowed"]
    health -->|"Degraded<br/>(< 10% free memory or disk)"| degraded["/health 200<br/>everything allowed, badge shown"]
    health -->|"Unhealthy<br/>(< 5% free, measured)"| unhealthy["/health 503<br/>deployments refused<br/>status, services, IIS, certificates,<br/>uninstall still work"]
    health -->|"check could not run<br/>(e.g. WMI)"| unknown["logged<br/>deployments not blocked"]
Hold "Alt" / "Option" to enable pan & zoom

An Unhealthy agent refuses to deploy BenefitManager versions — over HTTP (503 with the reason) and over Service Bus (a failed reply) — and nothing else, because the other commands are how an operator frees the machine.

User journey#

journey
    title An operator brings a new server under CommandCenter
    section Once per server
      Install agent: 3: Operations
      Server cert: 3: Operations
      Allow WebApi: 3: Operations
    section Register
      Type address: 5: Operator
      Agent names itself: 5: Operator
      Verified + health: 4: Operator
      Pick URL, accept: 5: Operator
    section Operate
      Fleet + deploys: 5: Operator
      Maintenance, IIS: 4: Operator
      Unhealthy, no deploy: 3: Operator
      Free disk, deploy: 4: Operator
    section Years later
      Certs renew alone: 5: Operator, Operations
Hold "Alt" / "Option" to enable pan & zoom

Once per server, operations installs the agent, gives it a server certificate by subject (the BCS root and intermediates are already on the machine) and allows WebApi's subject and the client CA in ClientAuthentication. From then on the operator works only on /config/agents: types the address, sees the agent name itself and its bindings, verifies, picks the server URL and accepts. Day to day the server shows up on the fleet dashboard and the deployment page; an Unhealthy agent refuses deployments with the reason until disk or memory is freed. Certificates renew without anyone doing anything.

Configuration#

WebApi

Setting Meaning
Agents:ClientCertificate:Source KeyVault (with KeyVaultUri, CertificateName; latest version, managed identity), Store (with Subject; newest valid with a private key) or File (with FilePath, Password; re-read when it changes). Empty: no certificate.
Agents:ClientCertificate:RefreshInterval How often it is re-read. Default 01:00:00.
Agents:ServiceBus:Namespace / ConnectionString The Service Bus channel; off while both are empty.
BenefitsAgent:ApiKey Deprecated. Sent as X-Api-Key only while set.

Agent (details in the agent guide)

Setting Meaning
ServerCertificate:Subject The agent's HTTPS certificate, chosen by subject from the store.
ClientAuthentication:AllowedSubjects WebApi's certificate subject. Required — empty accepts nothing.
ClientAuthentication:TrustedIssuers The client issuing CA (not the root).
ClientAuthentication:RevocationMode NoCheck for the BCS PKI until it publishes revocation (#311).
Authentication:ApiKey Deprecated; the scheme exists only while set.
Agent:Name, Agent:Port Reported name (default the machine name) and port (default 5010).

Rolling it out#

  1. Apply the migrations (GET /api/maintenance/migrate): AddAgentRegistry, AddAgentHeartbeats, AddHeartbeatInstallations, AddAgentReportedInfo.
  2. Issue WebApi's client certificate from the client CA with a fixed subject; set Agents__ClientCertificate__*.
  3. On each server, once: the agent's server certificate by subject, ClientAuthentication, and an agent release that has /api/agent/info, the health gate and the fail-closed policy.
  4. Add or verify the HTTP agents on /config/agents with https:// addresses; accept them.
  5. Remove the API key everywhere; the key once committed to the agent's settings (123-abcd) is public and must be replaced wherever it was used.