Skip to content

Developing against Azure#

The development stack has two stack targets. You choose one in run-dev-enhanced.ps1 (main menu 7).

Local (offline) Azure
The apps (gateway, main API, WebApi, SPA, OPA, Keycloak) Docker, with dotnet watch the same
Shared configuration App Configuration emulator your own App Configuration store
Key Vault none your own vault
Blob storage (host status, customer data, packages) Azurite your own storage account
SQL the local mssql container your own Azure SQL database
Service Bus (agents) off the shared namespace
Log Analytics, App Insights none shared workspace, your own App Insights
Needs Docker Docker, network, az login once a day for the firewall rule

Local is the stack as it always was, and it stays the fallback when you are offline. Azure takes SQL Server and the emulators off your machine, and gives you what cannot be emulated: - Key Vault references; - Log Analytics for the Exceptions page; - Service Bus request/reply with agents.

The data in the two targets is separate, and nothing is copied between them.

flowchart LR
    subgraph Laptop["Your machine (Docker)"]
        BFF["Gateway"] --- API["Main API"] --- WebApi
        OPA["OPA"] --- Keycloak
    end
    subgraph Own["rg-cc-dev-{alias} (yours)"]
        AC["App Configuration<br/>Developer tier"]
        KV["Key Vault"]
        ST["Storage<br/>config-status, customer-data, packages"]
        AI["App Insights"]
    end
    subgraph Shared["rg-cc-dev-shared"]
        SQL[("Azure SQL<br/>cc-dev-{alias}")]
        SB["Service Bus<br/>agent topics"]
        LA["Log Analytics"]
    end
    Laptop -- "signed in as cc-dev-{alias}<br/>(certificate)" --> Own
    Laptop -- "token, no password" --> SQL
    WebApi --> SB
    WebApi --> LA
    AI --> LA
Hold "Alt" / "Option" to enable pan & zoom

Who runs what#

Script Who When
scripts/dev-azure/provision-shared.ps1 an admin once, and after a change to infra/dev/shared.bicep
scripts/dev-azure/onboard-developer.ps1 an admin, who must be a member and an owner of the SQL administrators group once per developer
scripts/dev-azure/connect.ps1 the developer, with their own az login once on each machine, and to refresh
scripts/dev-azure/offboard-developer.ps1 an admin when a developer leaves

All four need PowerShell 7 and the Azure CLI. Onboarding also needs go-sqlcmd (winget install sqlcmd), because the older ODBC sqlcmd cannot sign in with an Azure CLI login.

The order is always the same: the shared services exist first, then each developer is onboarded, then each developer connects their own machine.

flowchart LR
    P["provision-shared.ps1<br/>admin, once"] --> O["onboard-developer.ps1<br/>admin, per developer"]
    O --> C["connect.ps1<br/>developer, per machine"]
    C --> R["run-dev-enhanced.ps1<br/>7 → Azure, 1 → 1"]
    O -. "when a developer leaves" .-> F["offboard-developer.ps1"]
Hold "Alt" / "Option" to enable pan & zoom

Provisioning the shared services#

An admin does this once per subscription, before the first developer is onboarded. It creates rg-cc-dev-shared with everything that has a base charge and is shared by all developers.

Before you start#

  • A subscription for development. The SQL free offer counts per subscription (10 databases), so a subscription of its own keeps those for the developers.
  • Your permissions.
    • To provision: Contributor on the subscription, to create the resource group and its resources, plus User Access Administrator (or Owner), because the template defines a custom role.
    • To onboard later: the same, plus permission in Entra ID to create app registrations. That is either the Application Developer role, or the tenant setting that lets users register applications.
  • An Entra group for the SQL administrators. It becomes the SQL server's administrator (the server accepts Entra sign-in only, so there is no SQL login or password). You don't need to make it yourself: the script suggests cc-dev-sql-admins and creates it when it does not exist, with you as its owner and member, after asking (-CreateGroup skips the question). Creating it needs the Entra setting that lets users create security groups, or Groups Administrator.
    • Every developer becomes a SQL administrator, as on the other database servers where developers are sysadmin: onboarding adds them to the group. -NoSqlAdmin onboards a developer with their own database only.
    • The containers never are: they sign in as the developer's service principal, which only has its own database. So a mistake in code running on the stack cannot reach another developer's database.
    • Whoever runs onboarding must be a member (it creates database users) and an owner of the group (it adds the developer). Provisioning makes its runner both.

  • The resource providers registered on the subscription. A new subscription may not have them yet; registering is harmless when they are already there:

    foreach ($ns in 'Microsoft.ServiceBus', 'Microsoft.Sql', 'Microsoft.OperationalInsights', 'Microsoft.Insights',
                'Microsoft.AppConfiguration', 'Microsoft.KeyVault', 'Microsoft.Storage') {
    az provider register --namespace $ns --wait
}

  • Signed in with az login, to the tenant that holds the subscription.

Running it#

./scripts/dev-azure/provision-shared.ps1 -Subscription "<subscription name or id>"
Parameter
-Subscription The subscription, by name or id. The script switches the Azure CLI to it.
-SqlAdminGroup The Entra group that administers the SQL server, by name or object id. cc-dev-sql-admins by default; created when missing, after asking.
-CreateGroup Create a missing group without asking.
-Location The Azure region, westeurope by default. Onboarding has its own -Location: keep it the same, because a developer's database must be in the SQL server's region.
-WhatIf Shows what would change (az deployment group what-if) and changes nothing. It needs rg-cc-dev-shared to exist, so it is for later runs, not the first one.

It deploys infra/dev/shared.bicep as the deployment cc-dev-shared. Onboarding and connect.ps1 read that deployment's outputs to find the shared resources. What it creates:

Resource Name Notes
Resource group rg-cc-dev-shared tagged purpose=commandcenter-dev
Service Bus namespace (Standard) sb-cc-dev-{token} local auth off (Entra only), TLS 1.2; the topics agent.state.v1, agent.commands.v1, agent.replies.v1, deploy.replies.v1, bmo.deploy.commands.v1, bmo.package.events.v1, messages kept for a day. Subscriptions are per developer (onboarding) or made by WebApi itself
Entra group cc-dev-sql-admins (or -SqlAdminGroup) only when missing; you as owner and member
Azure SQL logical server sql-cc-dev-{token} Entra-only authentication, administered by the group, TLS 1.2, public endpoint with no firewall rules: each developer's connect.ps1 adds a rule for their own IP, and onboarding adds a temporary one for the admin
Log Analytics workspace log-cc-dev-{token} 30 days' retention, capped at 1 GB a day. Each developer's App Insights writes here
Custom role CC Dev SQL Firewall (rg-cc-dev-shared) reads the SQL server and manages its firewall rules, nothing else. Onboarding gives it to each developer on the server

{token} is a short hash of the resource group's id, so the names are the same every time the script runs in that subscription. When it finishes, it prints the Service Bus and SQL server addresses and the Log Analytics workspace id.

No database is created here: each developer's database is created at onboarding.

Changing it later#

Edit infra/dev/shared.bicep, check with -WhatIf, then run the script again. It is idempotent: it updates the resources in place and keeps every developer's database, subscriptions and roles, which live in their own deployments (cc-dev-developer-{alias}).

Removing everything#

There is no script for this, on purpose:

  1. Offboard every developer first (below). That removes their resource groups, their databases in the shared server, and their app registrations.

  2. Then delete the shared resource group:

    az group delete -n rg-cc-dev-shared

    This deletes the Service Bus namespace, the SQL server and the Log Analytics workspace with everything in them. It cannot be undone.

Deleting rg-cc-dev-shared before offboarding leaves each developer's resource group and app registration behind, still billed where they have a charge.

Onboarding a developer#

sequenceDiagram
    actor Admin
    actor Dev as Developer
    participant Entra
    participant Azure
    participant SQL as Azure SQL
    Admin->>Entra: app registration + service principal cc-dev-{alias}, Dev as owner
    Admin->>Azure: rg-cc-dev-{alias}: store, vault, storage, App Insights, roles (developer.bicep)
    Admin->>Azure: rg-cc-dev-shared: database, heartbeat subscription, roles (developer-shared.bicep)
    Admin->>SQL: CREATE USER … FROM EXTERNAL PROVIDER WITH OBJECT_ID (service principal, Dev), db_owner
    Dev->>Azure: firewall rule dev-{alias} = this machine's IP
    Dev->>Dev: certificate + private key in %APPDATA%\CommandCenter\azure-dev
    Dev->>Entra: add the certificate's public part (as owner)
    Dev->>Azure: settings into the store (label dev), App Insights string into the vault
    Dev->>Dev: .env.azure
Hold "Alt" / "Option" to enable pan & zoom

  1. The admin runs:

    ./scripts/dev-azure/onboard-developer.ps1 -Subscription "<subscription>" -Alias <alias> -User <upn>

    The alias is 2–12 lowercase letters and digits. Pass the same -Location as for provisioning when it is not westeurope. The developer becomes a SQL administrator of the development server (a member of the group); add -NoSqlAdmin to give them their own database only. 2. The developer runs run-dev-enhanced.ps1, then 7 → 3, which calls connect.ps1, and enters the subscription and their alias. After that, 7 → 2 chooses Azure. 3. The developer starts the stack (1 → 1). The first time, they apply the migrations with 3 → 1, which runs GET /api/maintenance/migrate in both running APIs.

journey
    title A day in Azure mode
    section Start
      az login (once a day): 3: Developer
      run-dev-enhanced 1 → 1: 5: Developer
      Firewall rule follows your IP: 5: run-dev-enhanced
    section Work
      Edit code, dotnet watch reloads: 5: Developer
      Change a setting on /config (applies within 2 min): 4: Developer
    section Offline
      Azure not reachable, local stack offered: 4: run-dev-enhanced
Hold "Alt" / "Option" to enable pan & zoom

How the containers sign in#

Each developer has a service principal, cc-dev-{alias}. connect.ps1 makes its certificate on your machine: - The private key never leaves the machine. It stays in %APPDATA%\CommandCenter\azure-dev\cc-dev-{alias}.pem, readable by you only, and is mounted read-only into the containers as /run/secrets/cc-dev-cert. - Only the public part goes to the app registration, which you own. Adding it with --append keeps the certificate of another machine of yours. - The certificate is valid for a year. run-dev-enhanced.ps1 warns two weeks before it expires; connect.ps1 -RenewCertificate makes a new one.

Your own az login does not reach into the containers: on Windows the Azure CLI token cache is encrypted for your Windows account, so a Linux container cannot use it.

The hosts' own databases sign in with the same identity, through a token callback (AzureSqlSignIn). The connection strings have no user, no password and no Authentication=. See backend/CLAUDE.md → Per-stage connections.

What is where#

compose.azure.yaml holds only the bootstrap values, read from .env.azure: - which store (Configuration:AppConfig:Endpoint); - which vault (Configuration:KeyVault:Uri); - where host status goes (Configuration:Status:BlobServiceUri); - which identity signs in (Configuration:Identity:* and AZURE_*); - a snapshot file of its own (config-snapshot.azure.json), so a snapshot of the emulator's settings never starts a host in Azure mode.

It also blanks the emulator connection strings that compose.dev.yaml sets.

Everything else is in your store at label dev, where /config shows it and can change it. An environment variable would shadow the store (docs/configuration.md). connect.ps1 writes:

Key Value
ConnectionStrings:DefaultConnection, ConnectionStrings:CommandCenter your database, no credentials
CustomerData:Storage:ServiceUri, BmoBlob:ServiceUri, BmoBlob:Container your storage account, container packages
Agents:ServiceBus:Namespace, Agents:ServiceBus:StateSubscription the shared namespace, commandcenter-webapi-{alias}
LogAnalytics:WorkspaceId the shared workspace
APPLICATIONINSIGHTS_CONNECTION_STRING a Key Vault reference to appinsights-connection-string in your vault
CustomerData:Storage:ConnectionString, BmoBlob:ConnectionString, Agents:ServiceBus:ConnectionString empty, so a local value in your user secrets cannot win

User secrets sit below the store, so a key the store sets wins over them. connect.ps1 lists the connection-string keys in your user secrets that the store does not set, since those would still point at local resources. It prints their names only.

The store refreshes every 2 minutes in Azure mode, against 5 seconds with the emulator. Three hosts each check two sentinels, which comes to about 180 requests an hour, well inside the 3,000 a day the Developer tier includes. A change on /config therefore applies within 2 minutes.

What does not work in Azure mode#

  • Stage clone and database drop. They need disk BACKUP/RESTORE and master, which Azure SQL Database lacks. They are refused before anything changes: the clone at its first step, the drop task at its only step. DELETE /api/deploylogs/{id}/database answers 503 not-supported-on-azure-sql. Use the local stack for them.
  • Database sizes on the customers page and the metrics pages are unknown, because there is no sys.master_files.
  • Customer stages. Your Azure database starts empty, and stage rows that say localhost point at it. Stage databases in Azure are a later step.
  • BMO (profile bmo) stays on its local emulators. Its Service Bus subscription names are fixed, so two developers' BMOs would take each other's messages.
  • The first request after the database paused (an hour without the stack running) can fail while the database resumes, which takes up to a minute. Retry.

Costs#

Where Charge
Service Bus Standard shared a monthly base charge
Azure SQL shared server, a database per developer the free offer: 10 databases per subscription, 100,000 vCore-seconds a month each. The stack's pollers keep a database online while it runs, so it pauses only when the stack is down. Onboarding with -NoFreeOffer makes a billed serverless database instead
Log Analytics shared per GB ingested, capped at 1 GB a day
App Configuration Developer per developer a daily charge, with 3,000 requests a day included
Key Vault, storage, App Insights per developer pay per use, close to nothing

Offboarding#

./scripts/dev-azure/offboard-developer.ps1 -Subscription "<subscription>" -Alias <alias> -User <upn>

It lists everything onboarding created for that alias, and deletes exactly that after you type the alias: - the developer's resource group, then it purges their Key Vault so the name is free again; - their membership of the SQL administrators group; - their database, subscription and firewall rule; - only the role assignments onboarding granted; - the app registration.

It never deletes anything else. The developer removes .env.azure, .dev-stack.json and the certificate folder on their own machine.

Troubleshooting#

Symptom Cause, fix
Deployment 'cc-dev-shared' was not found in 'rg-cc-dev-shared' The shared services have not been provisioned in this subscription yet (provision-shared.ps1), or the Azure CLI is on another subscription
provision-shared.ps1 fails on the role definition Your account lacks User Access Administrator or Owner on the subscription
A deployment fails with MissingSubscriptionRegistration A resource provider is not registered: see Before you start
run-dev-enhanced offers the local stack Its checks listed what is wrong: no .env.azure, the certificate missing or expired, or the store not answering (offline)
A host stops at start with "cannot reach the store" Configuration:AppConfig:Optional is false in Azure mode, deliberately. Check the network, or start the local stack
Login failed on the database The firewall rule is not at your current IP: connect.ps1 -FirewallOnly, or restart through run-dev-enhanced
AADSTS700027 or a certificate error The certificate on the app registration is not the one on disk: connect.ps1 -RenewCertificate
A setting from your user secrets still applies connect.ps1 listed it: remove it there, or set it in your store