Skip to content

ADR 0007 — Agent registry, per-agent transport and server availability#
Status Accepted
Date 2026-09-24
Authors Ruben Knuijver
Supersedes —
Superseded by —
Related ADR 0005 (WebApi absorbs BMO's API) — this decides how its last Phase 3 slice (the /api/agents groups) and Phase 4's fleet work reach agents; MR !249 (router split)

Context#

The last BMO slice to port is the thirteen routes under /api/agents: agent list and installations, service control, IIS app pools, certificates, uninstall and package sync. They carry no SQL and no state — each is a call to one agent — but porting them exposed that the way CommandCenter reaches agents does not fit how agents are actually deployed:

  • Transport is chosen per process, but reachability is per agent. Production agents may be reachable over Service Bus only; QA/UAT agents answer over HTTP; development varies. BMO picks one transport for the whole service at startup (ServiceBusAgentTransportRouter or HttpAgentTransportRouter since !249). A mixed fleet cannot be served.
  • The agent list is static configuration, and there are two of them. WebApi reads Management:Agents, BMO reads CommandCenter:Agents; they name different machines, and the agent client factory looks names up case-sensitively in a fixed dictionary. The system of record is actually dbo.TblServer: every row with Type = 'BenefitManager' must have an agent, and BenefitManager hosting is meant to scale out without configuration changes.
  • The routers live in BMO's host. IAgentTransportRouter and both implementations sit in Benefits.CommandCenter, although the same split already exists in the client libraries for deployments (IAgentTransport: HttpAgentTransport in Benefits.Agent.Client, ServiceBusAgentTransport in Benefits.Agent.Client.ServiceBus) and for state (IAgentStateReader). The history records no reason; the Service Bus router simply needed the host's request/reply sender, fleet store and options.
  • "HTTP fallback" was never real. The old router's summary promised it; in Service Bus mode an agent without a recent heartbeat has always been refused (503).
  • TblServer is shared. The legacy CommandCenter writes it, the customer reverse proxy (CommandCenter.BenefitsReverseProxy, via a configuration payload produced outside this repository) routes on server availability, and BMO's customer-data import inserts into it. WebApi maps it but has never written it. Online means available, Active means allowed to be used; TblServerOfflineLog records (ServerKey, CreateDate) — when a server went offline, nothing more.

Decision#

1. A separate agent table, one row per agent#

A new dbo table (working name TblAgent), owned by WebApi's CommandCenterContext, holds what CommandCenter knows about an agent:

  • the name and hostname the agent reports, and a nullable link (ServerKey) to its TblServer row;
  • registration state — Pending, Accepted, Rejected — with who decided and when;
  • how it is reached: whether it is on Service Bus, an HTTP base URL (nullable), and the outcome and time of the last HTTP verification (§4);
  • the last heartbeat time, and the IIS site bindings it last reported (§3).

TblServer gets no agent columns. A self-registered agent nobody has accepted must not appear to the legacy CommandCenter or to the reverse proxy, and TblServer has required columns another codebase writes. The agent's identity is TblServer.Name once linked; an agent's reported name must equal it.

2. Registration: Service Bus agents announce themselves, HTTP agents are added by hand#

  • An agent's first heartbeat on the state topic creates a Pending row. An operator accepts or rejects it on a new /config/agents page; accepting links it to a TblServer row, creating one if needed (§3).
  • HTTP-only agents are added by hand on the same page, with their base URL.
  • Only Accepted agents are addressable by the operator endpoints.

3. TblServer is managed on a config page, and agents can fill in Url#

  • A /config/servers page lists and edits TblServer (name, Url, Type, Active) and shows TblServerOfflineLog per server.
  • Accepting an agent that has no TblServer row creates one with Type = 'BenefitManager' (the only agent type today). Url is editable, and can be proposed by the agent: the heartbeat gains an additive, optional list of the IIS site bindings it hosts, from which the page offers the Url. The operator confirms; the agent never overwrites TblServer on its own.

4. Transport is per agent, and HTTP fallback is only ever a verified fact#

  • The transport-neutral pieces move to Benefits.Agent.Client: IAgentTransportRouter, HttpAgentTransportRouter, an agent-directory interface ("which agents exist, and how is each reached") that replaces the factory's static dictionary, and a per-agent router that delegates to the right transport for each call.
  • The Service Bus pieces move to Benefits.Agent.Client.ServiceBus: ServiceBusAgentTransportRouter, the request/reply sender (today ServiceBusAgentRequester in BMO's host), library-owned options for timeouts and heartbeat age, and a small heartbeat-check interface. Hosts implement the directory and the heartbeat check; BMO keeps its configuration and in-memory fleet store, so its behaviour does not change.
  • An agent on Service Bus is always called over Service Bus. HTTP is used for it only when HTTP has been verified: the config page runs an HTTP ping from WebApi to the agent and records the result. Nothing is assumed from the URL being present.
  • Fallback happens only before anything is sent. If a Service Bus agent's heartbeat is stale and its HTTP reachability is verified, the router may use HTTP. It never retries over HTTP after a Service Bus send timed out: the command may already have run, and service restarts, uninstalls and IIS actions are not idempotent.

5. TblServer.Online is owned by a background job#

  • A scheduled job (the existing Quartz + JobLeaseService pattern, so one replica per occurrence) sets Online from agent availability — heartbeat age for Service Bus agents, a probe for HTTP agents — and maintenance: a manual switch, or a scheduled window, per server.
  • Each online → offline transition appends a TblServerOfflineLog row.
  • Active is never touched by the job, and management commands still reach servers whose Active is false: blocking customer traffic must not block repair.

Consequences#

  • The operator endpoints in WebApi go through the per-agent router, not the agent client factory directly as the inventory and services endpoints do today; those move over too.
  • Production is Service Bus only, so the BFF must not send /api/bmo/agents to WebApi in production until WebApi has Service Bus (step 5 below). Until then that route is a per-environment override, not shared configuration.
  • The heartbeat consumer in WebApi must be replica-safe: one shared subscription so each heartbeat is handled once, written as an idempotent update of the agent row. Request/reply over per-request subscriptions already works across replicas.
  • TblServer.Online gets a second writer unless the legacy one stops. Whatever sets Online in the legacy CommandCenter today has to be switched off when the job goes live, or the two will fight. That is a precondition for step 6, to be confirmed in the legacy codebase.
  • The reverse proxy's use of Active is unverified. In this repository it routes on ServerOnline and ServerVersionActive; whether its configuration producer filters on TblServer.Active must be checked where that producer lives.
  • WebApi's "Agent" HTTP client stops retrying writes (DisableForUnsafeHttpMethods), and writes get a 60 s budget instead of 5 s, matching BMO's Service Bus service-action budget. Timeouts answer 503, not 500.
  • TblServerOfflineLog stays as it is — no reason, no end time. If the offline history needs either, that is a separate, additive change.

Sequence#

One MR per step.

  1. ~~Split AgentTransportRouter in place~~ — done, !249.
  2. ~~This ADR~~ — done, !251.
  3. Libraries (done, !252; !253 makes the factory resolve through the directory): move the routers and the request/reply sender, add the directory and heartbeat interfaces and the per-agent router; BMO implements them from configuration and its fleet store. No behaviour change.
  4. WebApi registry (done, !254): the agent table and migration, a database-backed directory, /config/agents (accept/reject, add HTTP agents, HTTP verification) and /config/servers (TblServer + offline log).
  5. WebApi operator endpoints (done, !255): the thirteen /api/agents routes over the per-agent router, the retry and budget changes. BFF route per environment only.
  6. WebApi Service Bus, in three parts: heartbeats, self-registration and commands (done, !256); the Online job with maintenance (done, !257); IIS bindings in the heartbeat, offered as the server URL (#300). heartbeat consumer and self-registration, request/reply, IIS bindings in the heartbeat; the availability job for Online with maintenance switch and schedule. After this, /api/bmo/agents can move to WebApi everywhere.

Explicitly not doing#

HTTP fallback after a Service Bus send; assuming HTTP reachability from configuration; agents writing TblServer without an operator; agent columns on TblServer; letting Active block management commands.