Skip to content

Service Bus + Wolverine Scatter-Gather — Research Notes#

Status: Research only. This document captures current findings and open questions to baseline a future brainstorming session. No implementation decisions have been made.

Why#

Decouple the BenefitManager Orchestrator from its Agents. Today the Orchestrator iterates servers sequentially over HTTP and polls for completion. A topic-based scatter-gather would let one published command fan out to all live Agents simultaneously, each replying when done — the Orchestrator aggregates responses. Cleaner failure semantics, no per-call liveness probe, no 3-second polling loop.

Current state (as of 2026-04-30)#

Aspect Today
Transport HTTP only — IAgentClient typed HttpClient with Microsoft.Extensions.Http.Resilience
Fan-out Sequential foreach (serverName in servers) in DeploymentOrchestrator.cs:107 (blob) and :159 (HTTP)
Polling PollUntilCompleteAsync on GET /api/deploy/status/{jobId}, 3 s interval, 10 min timeout
Agent registry Static Dictionary<string, AgentDefinition> in OrchestratorOptions.Agents from appsettings.json
Liveness Per-call TCP Socket.Connect in AgentClientFactory.cs:55-72; failures return OfflineAgentClient
State reporting Pull-only — StateMonitorService runs locally every 5 min; Orchestrator reads via GET /api/installations
Messaging infra None. No MassTransit, Wolverine, RabbitMQ, Azure.Messaging.ServiceBus, MediatR, or NServiceBus
Auth Per-call ApiKey / mTLS / EntraId enforced by the Agent at RequireAuthorization()

Why Wolverine, not MassTransit#

MassTransit's license changed in 2024 (commercial requirements for v9+). It is excluded from this codebase.

Wolverine (JasperFx) is the leading OSS alternative: - In-process command bus with Azure Service Bus transport - First-class Saga support (long-running workflows correlated by ID) - Built-in scatter-gather and request/reply patterns

Other candidates for the design phase: - Rebus — OSS, lighter, no native Saga DSL but workable with custom state; friendlier AOT story - Raw Azure.Messaging.ServiceBus — smallest dependency footprint, most hand-rolled code, easiest AOT - NServiceBus — commercial, excluded unless budget is allocated

Proposed shape (sketch, not committed)#

Phase A — Passive state push (low risk, high value)#

Agent's StateMonitorService publishes to a Service Bus topic (agent.state.v1) every 5 min and on local state change. Orchestrator subscribes (one subscription per topic), maintains a hot in-memory view of each Agent's IInstallationRegistry. bmo status reads the local view instead of calling each Agent live.

Benefits: most of the decoupling value, doesn't touch the deploy hot path, fully reversible.

Phase B — Scatter-gather deploy#

Orchestrator publishes DeployRequested to topic deploy.commands.v1 carrying the blob URI (small payload — already used in the blob-reference deploy path). Each Agent has a topic subscription with a SQL filter on targetServer. Agents reply on deploy.replies.v1 (or via Wolverine request/reply correlation). Orchestrator runs a Saga keyed on JobId, aggregating replies until all expected Agents respond or a timeout fires.

This replaces DeploymentOrchestrator.cs:107-188 (sequential foreach + polling).

Hard constraints and risks#

AOT / trim compatibility#

BenefitManager.Contracts and BenefitManager.Agent.Client are IsAotCompatible=true. Wolverine relies on runtime IL generation (Lamar) and is not AOT-safe. Messaging glue must live exclusively in Orchestrator + Agent host projects. Contracts must remain pure POCO + System.Text.Json source-gen. Agent has EnableTrimAnalyzer=true — Wolverine registration code will surface IL2026/IL3050 warnings that need assessment.

Auth model shift#

Today every Agent enforces per-call RequireAuthorization() with ApiKey/mTLS/EntraId. With Service Bus, transport-level AuthN moves to Managed Identity on the Service Bus namespace. Per-call client-cert enforcement is no longer possible; Agents must authorize on message-envelope claims or trust the namespace ACL.

Operational cost and availability#

Service Bus Standard tier is required for Topics. Adds a runtime dependency — when SB is unavailable, deploys halt. Today an HTTP failure is per-Agent and isolated. Cost model needs evaluation against actual message volume.

Secret management#

appsettings.json already commits a real Azure Storage AccountKey in plaintext. Adding a Service Bus connection string the same way compounds the problem. Managed Identity is the right approach here.

Agent registry semantics#

Today the registry is a static config map. With pub/sub, "which agents exist" becomes derivable from active subscriptions or heartbeat messages. Need to decide whether the static map remains the source of truth during a transition.

At-least-once delivery#

Service Bus delivers at-least-once. Deploy commands need idempotency on the Agent side. This is partially addressed today via JobId correlation; a fuller idempotency story is required before Phase B.

Open questions for the design session#

  • Scope: Phase A only, Phase A then B, or full replacement of HTTP?
  • Library: Wolverine vs Rebus vs raw Azure.Messaging.ServiceBus? AOT pressure favors "raw" or Rebus; Saga ergonomics favor Wolverine.
  • Topic topology: Topic-per-message-type vs single topic with SQL filters?
  • Orchestrator HA: Today it is a CLI invocation (ephemeral). A Saga-based workflow implies a long-running Orchestrator process or an external workflow store. How is this hosted?
  • Migration path: Keep HTTP transport in parallel during rollout, or hard-switch?
  • Cost model: Estimate Service Bus pricing under the expected deploy + state-update message volume.

Run a dedicated brainstorming session using superpowers:brainstorming with this document as the baseline. Tackle after bmo package push metadata-flags work ships.