Skip to content

Global architecture and production-readiness brief#

Status: Architecture baseline and refactor proposal Decision: Retain the control-plane/agent model; harden the seams and make deployment state durable before production rollout Scope: bmo, React dashboard, CommandCenter, Service Bus, blob package store, and Benefits.Agent

Executive assessment#

The repository has the correct strategic shape for a fleet deployment platform: clients submit intent, CommandCenter owns business state and coordination, and agents perform host-local changes. The main production risk is not component selection; it is inconsistency at the boundaries:

  • deployment coordination is partly durable and partly process-local;
  • Service Bus, blob, and HTTP paths can expose different operational behavior;
  • agent snapshots and SQL state can be confused when snapshots are stale;
  • package integrity, transport authentication, and rollback guarantees need explicit enforcement;
  • the frontend currently includes a development server/mock surface and must not be treated as the production control plane.

Production gate: Do not promote the platform to production until all P0 items in the backlog have an owner, automated verification, and an operational runbook.

The Benefits platform uses a thin-client and control-plane architecture. The CLI and dashboard submit intent; CommandCenter owns policy, orchestration state, and SQL finalization; agents execute desired-state commands on Windows hosts.

Ownership boundaries#

Boundary Owns Does not own
bmo CLI and React dashboard User interaction, request composition, status presentation Deployment policy, SQL finalization, direct database access
CommandCenter Desired environment state, policy checks, deployment coordination, correlation IDs, final outcomes Local files, IIS, or Windows Service mutation
Service Bus and blob store Durable asynchronous transport and package distribution Business decisions or local installation state
Agent Installed files, IIS/services, local health, agent snapshots, idempotent execution Customer/stage truth or SQL activation decisions
CommandCenter SQL Environment, customer/stage, package, and deployment records Host-local runtime health

Agent snapshots are authoritative for what is installed and available on a host. They are observations consumed by CommandCenter; they do not replace the desired-state and business records in SQL.

Current state versus target state#

Concern Current state Target production state Gap
Client boundary CLI and dashboard call CommandCenter, but the frontend includes a separate mock Express server and development-oriented assumptions CLI and UI are thin clients of authenticated CommandCenter APIs; mock server is development-only and clearly isolated High
Deployment coordination Service Bus consumer fans out work and finalizes successful servers, but deployment job/fan-out state is not fully durable across CommandCenter restart A durable deployment aggregate stores correlation ID, target state, retries, partial failures, and terminal outcome Critical
Transport AgentTransportRouter supports Service Bus and HTTP fallback; fallback policy and selected transport are not consistently visible to operators Service Bus/blob is default; HTTP is explicit, authenticated, observable fallback with the same command/status contract High
Package distribution Blob references and SHA-256 metadata exist; package acceptance and agent-side validation must be enforced consistently Immutable, verified manifests; integrity checked before extraction; optional signing and provenance policy for production Critical
Agent state Fleet snapshots provide local observations and are age-checked for some operations Desired state remains in SQL; local state has freshness, sequence/version, and reconciliation semantics High
Failure recovery Partial success is logged; rollback/compensation is not a defined deployment contract Each step has a compensating action or an explicit operator remediation path Critical
Queues and workers Some workers use bounded channels; in-memory stores and process-local queues lose work/status on restart Durable job intake/outbox, bounded concurrency, retry policy, dead-letter handling, and restart recovery Critical
Authentication Frontend hardcoded credential was removed; authentication still depends on hosting configuration Browser uses a user/session-safe flow or same-origin BFF; service-to-service uses mTLS/Entra identity or rotated secrets Critical
Auditability Structured logs and audit UI exist Every command has actor, correlation ID, target, package digest, transport, decision, and terminal result retained High
Operations Local Docker stack and Windows services are documented Production runbooks cover rollout, key/certificate rotation, queue drain, replay, rollback, and disaster recovery High

Runtime topology#

flowchart LR
    User[Operator / DevOps]
    subgraph Client[Client]
        CLI[bmo CLI<br/>Benefits.Orchestrator]
        UI[React Dashboard<br/>frontend]
    end
    User --> CLI
    User --> UI

    subgraph Control[Control plane]
        CC[CommandCenter<br/>Minimal API + services]
        SQL[(CommandCenter SQL)]
        CC --> SQL
    end
    CLI --> CC
    UI --> CC

    subgraph Bridge[Bridge services]
        SB[Service Bus / event bus]
        Store[(Blob package store)]
    end
    CC <--> SB
    CC <--> Store

    subgraph AgentHost[Agent hosts]
        Agents[Benefits.Agent instances]
        State[Local state + snapshots]
        IIS[Windows IIS]
        Services[Windows Services]
        Agents <--> State
        Agents --> IIS
        Agents --> Services
    end
    SB <--> Agents
    Store -. package reference .-> Agents
    CC -. explicit HTTP fallback .-> Agents
Hold "Alt" / "Option" to enable pan & zoom

Service Bus plus blob references are the preferred bridge for deployment and agent management. HTTP is retained for development, recovery, and environments without Service Bus, but it must use the same CommandCenter routing and status contracts rather than creating a second orchestration implementation.

Deployment saga#

sequenceDiagram
    participant C as CLI / UI
    participant CC as CommandCenter
    participant SQL as CommandCenter SQL
    participant Store as Blob store
    participant SB as Service Bus
    participant A as Agent

    C->>CC: Submit deployment intent
    CC->>SQL: Persist job and correlation ID
    CC->>Store: Resolve and validate package manifest
    CC->>SB: Publish desired-state command
    SB->>A: Deliver package reference and command
    A->>Store: Download package
    A->>A: Validate, install, configure IIS/services
    A-->>SB: Correlated result and local snapshot
    SB-->>CC: Agent result
    CC->>SQL: Finalize successes and record failures
    CC-->>C: Queryable deployment status
Hold "Alt" / "Option" to enable pan & zoom

Each target server has an independent result. A partial deployment is a first-class outcome: successful targets can be finalized while failed targets remain visible for retry or remediation. Agent commands must be idempotent by correlation/job ID, and late or duplicate replies must not apply finalization twice.

State ownership and reconciliation#

flowchart TB
    Desired[Desired environment state<br/>stages, policy, package, deployment outcome] --> CC[CommandCenter + SQL]
    Local[Installed files<br/>IIS/services<br/>local capabilities] --> Agent[Agent host]
    Agent -->|snapshot and events| CC
    CC -->|desired-state commands| Agent
    CC -. does not infer business truth solely from .-> Local
Hold "Alt" / "Option" to enable pan & zoom

CommandCenter should treat stale or missing snapshots as an operational condition, not as proof that a version is absent. Management commands must surface the stale state and use an explicit retry or HTTP fallback policy.

Current implementation alignment#

The repository already provides the main building blocks:

  • AgentTransportRouter selects Service Bus or HTTP for management operations.
  • BmoDeployConsumer fans out blob-reference deployments and finalizes successful servers.
  • AgentStateConsumer and fleet state snapshots report host-local observations.
  • PackageReference and package manifests allow agents to download artifacts without receiving the package through every control-plane request.
  • IdempotencyStore prevents duplicate Service Bus deployment execution on an agent.

The remaining alignment work is operational rather than conceptual:

  1. Persist CommandCenter deployment jobs and fan-out results so restarts do not lose status.
  2. Make transport selection, fallback, and stale-snapshot decisions visible in job/audit records.
  3. Keep all client authentication configured by the hosting environment; never ship a credential in frontend source.
  4. Apply bounded capacity and explicit retry/dead-letter behavior consistently to deployment and customer-data workers. Customer-data export intake is bounded to ten queued jobs and returns 503 Service Unavailable when capacity is not available within the enqueue window.
  5. Keep package lifecycle, fleet inventory, and customer/stage workflows as separate CommandCenter domains with shared contracts only where necessary.

Production hardening controls#

These controls are required to convert the architecture from a sound design into a production-safe platform:

  1. Identity and transport: require HTTPS for every HTTP hop, validate server and client certificates where mTLS is selected, prefer workload identity for Azure resources, and rotate any remaining secrets through a managed secret store.
  2. Package integrity: validate the manifest, digest, size, version, and package status in CommandCenter; verify the digest again at the agent immediately before extraction; reject invalid archive paths and packages that fail policy.
  3. Durable coordination: persist the deployment aggregate before publishing commands; use an outbox or equivalent transactional handoff so a SQL commit cannot succeed while the message is lost.
  4. Idempotency: use a stable deployment/target correlation key for retries, agent execution, replies, and finalization. Duplicate messages must be safe and observable.
  5. Backpressure: bound every in-process queue, return an explicit overload response, and expose queue depth/oldest age. Do not rely on client timeouts to control server work.
  6. Failure recovery: define whether a failed step is retried, compensated, or escalated. Never report a global success when target-level results are incomplete.
  7. Observability: emit structured events for request, policy decision, publish, delivery, execution, finalization, retry, dead-letter, and operator remediation. Avoid logging package contents or customer PII.
  8. Least privilege: CommandCenter should have only the SQL and storage permissions it needs; agents should access only their assigned package scope and host resources.
  9. Browser safety: do not compile CommandCenter API keys into Vite bundles. Use same-origin session authentication, a BFF, or another browser-safe identity flow.
  10. Recovery testing: regularly test CommandCenter restart, Service Bus redelivery, duplicate replies, stale snapshots, blob unavailability, agent restart during extraction, and partial fleet failure.

Implementation gaps and architectural risks#

The following gaps are the concrete implementation work behind the current-vs-target comparison. They are ordered by the consequence of leaving the gap unresolved, not by implementation effort.

Gap Current evidence Risk Refactor action Planned verification
Non-durable deployment coordination BmoDeployConsumer performs fan-out and finalization, while process-local execution state can disappear during restart Lost or ambiguous deployment outcomes; duplicate operator actions Introduce durable deployment/job and target-result records plus an outbox/inbox handoff Restart during each saga phase; replay and duplicate-message tests
Incomplete package trust boundary Package references and SHA-256 metadata exist, but verification must be guaranteed at the agent immediately before extraction Tampered package or unsafe archive can mutate a host Enforce manifest/status/digest checks in CommandCenter and repeat digest/archive validation in Agent Wrong digest, rejected status, traversal archive, truncated blob tests
Split transport semantics AgentTransportRouter supports Service Bus and HTTP, but timeout, fallback, and audit semantics are not one durable policy Operators see inconsistent behavior and may retry unsafe operations Create one command envelope and transport policy; record selected path and fallback reason HTTP/SB parity, timeout, offline-agent, and fallback tests
Weak failure compensation contract Successful targets can be finalized while failed targets are logged, but rollback/repair is not modeled as a state machine Hosts can remain partially mutated while SQL presents an incomplete picture Add target states such as Succeeded, Failed, Retryable, and NeedsRemediation; define compensations per step Failure injection after extraction, IIS update, and service configuration
Stale local observations Fleet snapshots are age-checked in selected paths, but freshness and reconciliation are not a universal contract Missing/stale state can be mistaken for absence or health Add snapshot sequence/freshness metadata and explicit stale responses to CLI/UI Stale heartbeat, out-of-order snapshot, agent restart, reconciliation tests
Process-local workflow state Export state and queues are held in memory, although export intake is now bounded Restart loses job status and may leave staged files or accepted requests Persist critical job metadata; make queue recovery and cleanup explicit Restart, cancellation, overload, and orphan-file tests
Authentication/configuration drift Frontend no longer ships the previous hardcoded key, but production identity enforcement is hosting-dependent Unauthorized access or accidental insecure deployment Fail production startup when required TLS/identity settings are absent; document browser-safe auth Negative startup/configuration tests and certificate rotation drill
Incomplete operational evidence Logs exist, but actor, package digest, transport, policy decision, and target result are not guaranteed as one audit record Incidents cannot be reconstructed reliably Define a correlation-linked audit schema and retention policy Query an entire deployment from one correlation ID

Implementation sequence for the gaps#

  1. Stabilize contracts: define deployment aggregate states, command envelope fields, package verification failures, and stale-snapshot responses before changing transports.
  2. Make the saga durable: add persistence and outbox/inbox behavior, then move existing BmoDeployConsumer fan-out logic behind those contracts.
  3. Harden the agent boundary: validate package and archive inputs independently of CommandCenter and make execution idempotent per target correlation key.
  4. Unify transport behavior: route HTTP and Service Bus through the same command/result model; do not duplicate business logic for the fallback.
  5. Close recovery and observability: add compensation/remediation commands, durable audit records, metrics, and failure-drill runbooks.

Risk-ranked backlog#

Rank ID Severity Risk Recommended owner Definition of done
1 CC-01 P0 Critical Deployment status and target outcomes can be lost or become ambiguous after CommandCenter restart CommandCenter Durable deployment/job tables, correlated target rows, restart recovery test, query endpoint used by CLI/UI
2 AG-01 P0 Critical A tampered or unsafe package could be installed if validation is bypassed or archive paths are unsafe Agent + PackageStore Digest verified at agent boundary, manifest policy enforced, archive traversal tests pass, rejected packages are audited
3 SEC-01 P0 Critical Service-to-service credentials or cleartext transport could expose deployment authority Platform/security HTTPS/mTLS or workload identity enforced in production configuration; secret rotation runbook and negative tests
4 DEP-01 P0 Critical Partial host mutation has no guaranteed rollback or compensating operation Deployment domain Step-level compensation matrix, explicit Partial/NeedsRemediation state, operator replay/remediation flow
5 MSG-01 P0 Critical Message loss, duplicate delivery, or non-atomic publish can desynchronize SQL and agents Messaging + CommandCenter Outbox/inbox or equivalent, idempotency keys, dead-letter replay procedure, duplicate-message tests
6 OPS-01 P1 High Stale fleet snapshots can be interpreted as absence or healthy state Fleet domain Freshness/sequence metadata, stale-state API contract, UI/CLI warnings, reconciliation command
7 MSG-02 P1 High HTTP fallback and Service Bus paths can diverge in timeout and error semantics Agent transport One command contract, explicit fallback policy, transport recorded in audit/job state, parity tests
8 DATA-01 P1 High In-memory export/import status and process-local queues do not survive restart Customer-data domain Durable job metadata, bounded workers, cancellation/overload tests, cleanup and resume semantics
9 AUD-01 P1 High Operators cannot reconstruct who requested a deployment or why a target was finalized Observability Correlation-linked audit record with actor, package digest, policy result, transport, target outcomes
10 UI-01 P1 High Frontend mock server and development assumptions could be mistaken for production behavior Frontend/platform Production build uses CommandCenter only; mock server isolated behind explicit development profile
11 OPS-02 P2 Medium Package lifecycle, fleet inventory, and customer/stage workflows overlap in cleanup decisions Architecture owner Ownership matrix, domain contracts, lifecycle tests, no duplicate deletion authority
12 DR-01 P2 Medium Recovery objectives and queue-drain procedures are undocumented Operations RTO/RPO defined, restore/replay drill completed, runbooks versioned with release

Overlapping features and duplicated responsibilities#

The platform has several legitimate capabilities that currently touch the same concepts. The goal is not to remove those capabilities; it is to give each one a single owner and a narrow contract.

Overlap Why it overlaps Target owner Refactor boundary
Package lifecycle vs deployment lifecycle Package push, accept/reject, supersede, prune, and deploy eligibility all reference the same version Package domain owns artifact state; CommandCenter deployment domain owns activation Package status gates deployment, but package commands never finalize environment SQL and deployments never delete package artifacts
Fleet inventory vs installed-version queries Agent snapshots, inventory endpoints, and CLI installed/status commands all describe host state Fleet domain owns host observations; query service projects them for clients Normalize one FleetStateEntry/installation contract and derive CLI/UI views instead of maintaining parallel interpretations
Deployment finalization vs lifecycle cleanup Finalization registers active versions; lifecycle removes unused versions Deployment domain owns activation; lifecycle domain owns candidate selection and removal request Lifecycle can request removal only after querying authoritative stage and deployment state; it cannot directly infer safety from an agent snapshot
Direct HTTP management vs Service Bus commands Both can remove versions, control services/IIS, manage certificates, and query agents IAgentTransportRouter is the single CommandCenter boundary Management endpoints submit a command; router chooses transport; transport-specific clients contain no business policy
Customer-data import/export vs deployment workflows Both use streaming, staged files, status polling, cleanup, and background workers Customer-data domain owns transfer jobs; deployment domain owns deployment jobs Share infrastructure primitives for job status, cleanup, and bounded queues only through generic contracts; keep data mapping and deployment policy separate
CLI and dashboard deployment flows Both can submit and monitor deployments CommandCenter owns the workflow; clients only compose requests and render status Use the same create/status/retry/remediation API and correlation model; no client-side orchestration
Audit log vs operational logging Deploy logs, structured logs, and audit UI all describe activity Audit domain owns durable operator/business events; logging owns diagnostics Emit one correlation ID into both, but do not use transient logs as the audit source of truth

Refactor rules for overlaps#

  • One policy owner: package acceptance, target selection, SQL finalization, and removal safety decisions belong in CommandCenter domain services.
  • One execution boundary: all host mutation goes through the Agent command contract, regardless of HTTP or Service Bus transport.
  • One state projection: clients consume CommandCenter projections; they do not merge SQL, snapshots, and local assumptions independently.
  • One lifecycle authority per resource: package retention, deployment activation, and customer/stage cleanup must have separate commands and permissions.
  • Shared mechanics, separate domains: queueing, correlation, retry, and cleanup helpers may be shared; domain status transitions and authorization must not be silently shared.

Overlap refactor deliverables#

  1. Produce a responsibility matrix for every CommandCenter endpoint and map each endpoint to one domain owner.
  2. Consolidate duplicate version/installation projections behind a single query service and contract.
  3. Replace client-specific deployment orchestration with a shared CommandCenter job API.
  4. Add authorization tests proving package operators cannot finalize deployments and lifecycle workers cannot bypass removal safety checks.
  5. Remove or isolate mock/development feature paths from production startup and deployment documentation.

Assuming a two-week sprint with one CommandCenter engineer, one agent/messaging engineer, and shared security/QA support:

Sprint goal#

Make one deployment fully traceable and restart-safe from client request through agent execution and SQL finalization, while closing the highest-risk package and transport controls.

Sequence#

  1. Baseline and contracts (days 1-2) - Freeze the deployment state machine and target-level outcome vocabulary. - Add a correlation/causation contract shared by CLI, UI, CommandCenter, Service Bus, and Agent. - Define the package verification contract: version, manifest digest, size, status, and extraction safety. - Add architecture decision records for transport fallback and state ownership.

  2. Durable deployment coordination (days 2-6) - Add deployment job and target-result persistence using the existing Dapper/data-access conventions. - Persist before publish; implement an outbox or equivalent retryable publish record. - Make BmoDeployConsumer update target state idempotently and expose a status endpoint consumed by both clients. - Add restart, duplicate reply, timeout, and partial-fan-out tests.

  3. Agent and package hardening (days 4-8) - Enforce digest verification immediately before extraction. - Validate version/environment/path inputs at the agent boundary and reject unsafe archive entries. - Define extraction cleanup and compensation behavior for failures after filesystem/IIS mutation. - Verify package-store permissions and production TLS/identity settings.

  4. Transport and operational resilience (days 7-10) - Make Service Bus/blob the default production route. - Make HTTP fallback opt-in or policy-driven, record the selected transport, and align timeouts/retries. - Add queue depth, retry, dead-letter, stale-agent, and deployment-age metrics. - Convert remaining process-local critical state to durable state or explicitly document it as non-production.

  5. Client and release readiness (days 9-10) - Ensure the frontend production profile has no mock server and no compiled service credential. - Update CLI/UI to show target-level status, stale state, transport, and remediation actions. - Complete threat-model review, runbook review, and a controlled failure drill.

Explicit overlap work within the sprint#

  • During days 1-2, create the endpoint-to-domain responsibility matrix and freeze the shared deployment/job contracts.
  • During days 3-6, move both CLI and dashboard deployment requests onto the same durable CommandCenter job/status API.
  • During days 6-8, make IAgentTransportRouter the only management execution boundary and add HTTP/Service Bus parity tests.
  • During days 8-9, separate package retention, activation/finalization, and lifecycle-removal permissions and add authorization tests.
  • During days 9-10, verify that UI/CLI projections use CommandCenter-owned status rather than independently combining agent and SQL data.

Sprint exit criteria#

  • A deployment can be submitted, restarted mid-flight, resumed, and queried without losing target outcomes.
  • Duplicate Service Bus messages and replies do not duplicate installation or SQL finalization.
  • Invalid/tampered packages are rejected by the agent before extraction.
  • Production configuration cannot start with cleartext service transport or missing required identity settings.
  • Partial success is visible as partial, never silently promoted to global success.
  • P0 backlog items have automated tests and operational owners; unresolved P1 items have explicit risk acceptance.

Decision log#

Decision Rationale
Keep CommandCenter as the only business/workflow coordinator Prevents CLI, UI, and agents from implementing competing deployment policy
Keep agents authoritative for local runtime observations only Avoids treating stale host state as customer/stage truth
Prefer Service Bus + blob references Reduces synchronous coupling and supports fan-out, retries, and package distribution
Retain HTTP as a controlled fallback Supports local development and recovery without creating a separate domain workflow
Treat partial deployment as a first-class result Fleet operations cannot assume atomic success across independent Windows hosts
Separate package lifecycle from deployment finalization Package acceptance is a supply-chain decision; activation is an environment-state decision