Skip to content

Plan: Multi-Agent Orchestrator Support#

Context#

The Orchestrator currently assumes a single Agent. AgentBaseUrl (one URL) and TargetVm (one hostname) are scalar strings baked into OrchestratorOptions. Each server in the SQL database TblServer represents a BenefitManager deployment target, but the Orchestrator can only talk to one. The legacy PowerShell deployed to ALL online servers; the C# replacement needs to match.

Goal: Replace the single-agent assumption with a config-driven agent dictionary. The Orchestrator should deploy to (and manage) multiple servers, each running its own Agent.

Pre-requisite: Bug Fixes#

Fix two pre-existing bugs before the refactoring.

0.1 SqlRemovalService.RemoveAsync — force flag ignored#

src/BenefitManager.Orchestrator/Services/SqlRemovalService.cs line 91: RemoveSql is hardcoded where the computed sql variable should be used. The --force flag never triggers DeactivateSql.

Fix: Change RemoveSql → sql on line 91.

0.2 DeployCommand — CancellationToken not passed#

src/BenefitManager.Orchestrator/Commands/DeployCommand.cs: orchestrator.RunAsync(...) is called without the cancellationToken, making Ctrl+C non-functional during deployment.

Fix: Pass cancellationToken through to RunAsync (it's already a parameter on the Spectre.Console ExecuteAsync method, just not forwarded).

Step 1: Configuration — Agents Dictionary#

Files to modify: - src/BenefitManager.Orchestrator/Configuration/OrchestratorOptions.cs - src/BenefitManager.Orchestrator/appsettings.json - src/BenefitManager.Orchestrator/appsettings.Production.json

Replace AgentBaseUrl + TargetVm with:

public Dictionary<string, AgentDefinition> Agents { get; init; } = new(StringComparer.OrdinalIgnoreCase);

New AgentDefinition class (same file or dedicated file): 

public sealed class AgentDefinition
{
    public string BaseUrl { get; init; } = string.Empty;
}

Dictionary key = server name (must match TblServer.Name). Auth settings (ApiKey, AuthMode, ClientCertificate, EntraId) remain global — all agents share the same auth.

Remove AgentBaseUrl and TargetVm properties.

Update appsettings.json: 

"Orchestrator": {
  "Agents": {
    "vmbenefitsbmp1": { "BaseUrl": "http://vmbenefitsbmp1:5100" }
  },
  "ApiKey": "",
  "PollingIntervalSeconds": 3,
  "PollingTimeoutMinutes": 10,
  "VersionsPath": "F:\\Versions",
  "AuthMode": "ApiKey",
  ...
}

Same pattern for appsettings.Production.json.

Step 2: IAgentClientFactory + DI Registration#

New files: - src/BenefitManager.Orchestrator/Services/IAgentClientFactory.cs - src/BenefitManager.Orchestrator/Services/AgentClientFactory.cs

public interface IAgentClientFactory
{
    IAgentClient GetClient(string serverName);
    IReadOnlyList<string> GetServerNames();
}

AgentClientFactory uses IHttpClientFactory.CreateClient("Agent") and sets BaseAddress per server. No caching — IHttpClientFactory manages handler lifetimes.

File to modify: src/BenefitManager.Orchestrator/Program.cs

Switch from typed client to named client:

// Before: services.AddHttpClient<IAgentClient, AgentClient>(...)
// After:
var httpClientBuilder = services.AddHttpClient("Agent", (sp, client) =>
{
    var opts = sp.GetRequiredService<IOptions<OrchestratorOptions>>().Value;
    if (opts.AuthMode.Equals("ApiKey", StringComparison.OrdinalIgnoreCase))
        client.DefaultRequestHeaders.Add("X-Api-Key", opts.ApiKey);
});
// mTLS / EntraId / resilience handlers remain on the builder (unchanged)
httpClientBuilder.AddStandardResilienceHandler();

services.AddSingleton<IAgentClientFactory, AgentClientFactory>();

IAgentClient and AgentClient themselves do not change — they remain a thin HttpClient wrapper.

Step 3: SqlRemovalService — Add targetServerName Parameter#

Files to modify: - src/BenefitManager.Orchestrator/Services/ISqlRemovalService.cs - src/BenefitManager.Orchestrator/Services/SqlRemovalService.cs

Change RemoveAsync signature to match FinalizeAsync's pattern:

// Before:
Task RemoveAsync(string environment, string version, bool force = false, CancellationToken cancellationToken = default);

// After:
Task RemoveAsync(string environment, string version, string targetServerName, bool force = false, CancellationToken cancellationToken = default);

In SqlRemovalService: use the new parameter instead of options.Value.TargetVm. Remove IOptions<OrchestratorOptions> from the constructor since it's no longer needed.

Step 4: DeploymentOrchestrator — Per-Server Fan-Out#

File to modify: src/BenefitManager.Orchestrator/DeploymentOrchestrator.cs

Constructor: replace IAgentClient agentClient with IAgentClientFactory agentClientFactory.

RunAsync gains an optional targetServers parameter:

public async Task<bool> RunAsync(
    string environment, string version, string zipFilePath,
    IReadOnlyList<string>? targetServers = null,
    CancellationToken cancellationToken = default)

Flow: 1. Resolve targets: targetServers ?? agentClientFactory.GetServerNames() 2. Phase 1: SQL pre-check (once, unchanged) 3. Phase 2+3: For each server sequentially — get client from factory, upload, poll. Log per-server status. On failure: log, skip to next server (continue-all semantics). 4. Phase 4: For each successful server — finalizationService.FinalizeAsync(env, version, serverName, ct) 5. Return false if any server failed, true if all succeeded.

Step 5: Commands — --server Option + Factory Injection#

Files to modify: - src/BenefitManager.Orchestrator/Commands/DeployCommand.cs - src/BenefitManager.Orchestrator/Commands/RemoveCommand.cs - src/BenefitManager.Orchestrator/Commands/ServiceCommand.cs - src/BenefitManager.Orchestrator/Commands/IisCommand.cs

Add to each *Settings class: 

[CommandOption("--server")]
[Description("Target specific server(s). Defaults to all configured agents.")]
public string[]? Servers { get; init; }

Resolution helper (used by all commands): 

IReadOnlyList<string> targets = settings.Servers is { Length: > 0 }
    ? settings.Servers
    : factory.GetServerNames();

DeployCommand#

Inject DeploymentOrchestrator (unchanged). Pass settings.Servers through to orchestrator.RunAsync(...).

RemoveCommand#

Replace IAgentClient with IAgentClientFactory. Loop over target servers: 1. SQL safety check (once): GetServerCountForVersionAsync 2. For each server: agentClient.RemoveAsync(env, version, force, ct) 3. For each server: sqlRemovalService.RemoveAsync(env, version, serverName, force, ct)

Safety check update: when serverCount <= targets.Count and stages exist, require --force.

ServiceCommand / IisCommand#

Replace IAgentClient with IAgentClientFactory. Loop over targets, call each agent's management endpoint.

Step 6: Test Updates#

File to modify: tests/BenefitManager.Orchestrator.Tests/DeploymentOrchestratorTests.cs

  • Mock IAgentClientFactory instead of IAgentClient.
  • agentClientFactory.Setup(f => f.GetServerNames()).Returns(["server1"]) for single-server tests.
  • agentClientFactory.Setup(f => f.GetClient("server1")).Returns(mockAgentClient.Object) to return the existing mock.
  • Add test: multi-server with both succeeding.
  • Add test: multi-server with one failing — verifies FinalizeAsync only called for the successful server.
  • Update SqlRemovalService mock setups for the new targetServerName parameter.

Files Summary#

Action File
Modify src/BenefitManager.Orchestrator/Configuration/OrchestratorOptions.cs
Modify src/BenefitManager.Orchestrator/Program.cs
Modify src/BenefitManager.Orchestrator/DeploymentOrchestrator.cs
Modify src/BenefitManager.Orchestrator/Services/ISqlRemovalService.cs
Modify src/BenefitManager.Orchestrator/Services/SqlRemovalService.cs
Modify src/BenefitManager.Orchestrator/Commands/DeployCommand.cs
Modify src/BenefitManager.Orchestrator/Commands/RemoveCommand.cs
Modify src/BenefitManager.Orchestrator/Commands/ServiceCommand.cs
Modify src/BenefitManager.Orchestrator/Commands/IisCommand.cs
Modify src/BenefitManager.Orchestrator/appsettings.json
Modify src/BenefitManager.Orchestrator/appsettings.Production.json
Modify tests/BenefitManager.Orchestrator.Tests/DeploymentOrchestratorTests.cs
New src/BenefitManager.Orchestrator/Services/IAgentClientFactory.cs
New src/BenefitManager.Orchestrator/Services/AgentClientFactory.cs

IAgentClient.cs and AgentClient.cs — unchanged.

Verification#

  1. dotnet build — 0 errors, 0 warnings
  2. dotnet test — all existing + new tests pass
  3. Single-agent config: Agents with one entry works identically to the old behavior
  4. Multi-agent config: Agents with two entries deploys to both sequentially; SQL creates TblServerVersion rows for each server
  5. --server filter: deploy Productie 25.5.29 app.zip --server vm1 targets only vm1
  6. Partial failure: if one agent is unreachable, the other proceeds; FinalizeAsync only called for the successful server; command returns exit code 1
  7. --force fix: remove Productie 25.5.29 --force executes DeactivateSql (not RemoveSql)