Skip to content

Logging levels and audit events#

Every CommandCenter service logs through Shared.Logging (AddLoggingConfiguration): to the console, to a rolling file, and — where it is configured — to Azure Monitor / Log Analytics or the local Aspire dashboard over OpenTelemetry.

For an investigation you can raise the level for a while — everything to Debug, or one area to Trace — without restarting anything. The change ends by itself at the time you give, and it can never silence audit events or the exception log.

How a level is decided#

flowchart LR
    Baseline["Logging:LogLevel<br/>(appsettings, per service)"]
    Runtime["Logging:Runtime<br/>default + overrides + expiry"]
    Pinned["Pinned: CommandCenter.Audit,<br/>the exception log (Information)"]
    Controller["LogLevelController"]
    MEL["Filter rules<br/>(every provider)"]
    Serilog["Serilog switches<br/>(console, file)"]
    OTel["OpenTelemetry<br/>(Azure Monitor, Aspire)"]

    Baseline --> Controller
    Runtime --> Controller
    Pinned --> Controller
    Controller --> MEL
    Controller --> Serilog
    MEL --> Serilog
    MEL --> OTel
Hold "Alt" / "Option" to enable pan & zoom
  • The baseline is each service's Logging:LogLevel (usually Default: Information, Microsoft.AspNetCore: Warning).
  • The runtime levels (Logging:Runtime) add a new default and per-category overrides, only until ExpiresAt. The most specific category wins, so a new default of Debug does not make Microsoft.AspNetCore chatty; give it an override if you want that too.
  • Pinned categories always record Information and above: audit events (CommandCenter.Audit) and the exception log entries the Log Analytics Exceptions page reads. No runtime change and no provider-specific setting can lower them.
  • One controller turns all of this into the filter rules every provider uses, and moves Serilog's level switches to match, so the console, the file and Azure Monitor always agree.

Setting a runtime level (/config/logging)#

flowchart LR
    SPA["/config/logging"] -- "PUT /api/logging/runtime?target=prod/webapi" --> WebApi
    WebApi -- "loglevel.write per category" --> OPA
    WebApi -- "Logging:Runtime at label prod/webapi,<br/>then Settings:Sentinel (prod)" --> AC["App Configuration"]
    AC -. "reload" .-> Hosts["every WebApi instance of prod"]
    Hosts -- "audit 900 applied … 901 ended" --> Logs["Log Analytics"]
Hold "Alt" / "Option" to enable pan & zoom

The Logging page sets the levels per target, an App Configuration label the hosts of one kind read:

Target Label Read by
Everywhere platform every host of every environment, the gateway included
Gateway platform/bff the gateway
Environment {env} every service of the environment
A service {env}/{service} that service in the environment
  • One value per target: a default level (or the baseline's), category overrides, for 15 minutes to 24 hours, and a reason. The page stores it as one JSON value at Logging:Runtime (below), with who set it, and moves the sentinel of the hosts that read the target (Settings:Sentinel for an environment or service target, Settings:PlatformSentinel for the gateway and everywhere); they apply it within a refresh.
  • Targets combine: a service reads everywhere, then its environment, then its own target. The more specific target wins field by field and overrides add up — the page says when more than one target is raised.
  • It ends by itself at the expiry; End now removes it at once (and Clear removes one that has expired). A change made from an older read is refused, so two operators cannot overwrite each other.
  • Audit categories cannot be changed: an override of CommandCenter.Audit (or the exception log) is refused by the page and by the policy (protected_log_category); a default of None still leaves them at Information.
  • config.read to see, loglevel.write to change (operators and configuration admins). WebApi records who set or ended what (audit 3187, 3188); each host records applying and ending (900, 901). The host that serves the page needs App Configuration Data Owner.

The value as stored (or set in appsettings.json locally, which reloads too):

"Logging": {
  "Runtime": {
    "Default": "Debug",
    "Overrides": { "CommandCenter.WebApi.Agents": "Trace" },
    "ExpiresAt": "2026-09-28T16:00:00Z",
    "Reason": "ticket 1234: agent heartbeats",
    "SetBy": "ruben"
  }
}
  • ExpiresAt is required. Without it, or once it has passed, nothing changes — a raised level can never be left on by accident. When it passes, the baseline applies again on its own, within a second, with no configuration change.
  • Levels: Trace, Debug, Information, Warning, Error, Critical, None. An override names a category prefix (a namespace or a class).
sequenceDiagram
    participant Operator
    participant Page as /config/logging
    participant Config as App Configuration
    participant Service
    Operator->>Page: prod/webapi: Debug for 1 hour, "ticket 1234"
    Page->>Config: Logging:Runtime (until 11:00), sentinel
    Config-->>Service: reload
    Service->>Service: rules + switches changed
    Service-->>Operator: audit 900 "Runtime log levels applied … until 11:00"
    Note over Service: 11:00
    Service->>Service: baseline again
    Service-->>Operator: audit 901 "Runtime log levels ended"
Hold "Alt" / "Option" to enable pan & zoom

Audit events#

An audit event says who changed, deleted or switched what. They are written with IAuditLogger (category CommandCenter.Audit) and keep their service's EventIds, so existing queries still match:

EventId Event
900, 901 Runtime log levels applied / ended
3081 A superseded deployment database dropped by the cleanup job
3111 A customer stage deleted
3113 A stage database dropped by a background task
3116, 3118 Customer data exported / imported
3132 A customer stage deactivated
3150 A server feature switched
3170–3176 Agents accepted or rejected; servers created or updated; maintenance set, ended or removed
3180, 3181 A setting changed / removed on a /config settings page
3182, 3183 A restart asked / migrations applied on the Hosts panel (928 when an instance restarts)
3184 An environment added, changed or removed in the catalogue
3185, 3186 A SQL Server registered, changed or removed / tested
3187, 3188 Runtime log levels set / ended on the Logging page
3189 The role map changed on the Access page
3190, 3191 Who may do what changed / reset to the release defaults on the Access page
3192, 3193, 3194 A user-token signing key added / activated / retired on the Credentials page
3195, 3196 The identity provider changed / reset to the deployment's settings on the Identity provider page
3197 Agent settings changed on the Configuration tab of /config/agents (the label and the keys)
3198, 3199 An agent restarted on request / refused to restart (with its reason)

To find them in Log Analytics: AppTraces | where Properties.CategoryName == "CommandCenter.Audit".

For the developer#

  • Log an audit event through IAuditLogger (from DI) with a [LoggerMessage] method taking this IAuditLogger logger, in your service's EventId range. Use it for changes a person makes or destructive work a task does, never for routine activity.
  • Tests pass NullAuditLogger.Instance.
  • Do not add a Serilog:MinimumLevel section: levels come from Logging, and a Serilog minimum would be overridden by the switches anyway.