Logging levels and audit events

Every CommandCenter service logs through Shared.Logging (AddLoggingConfiguration): to the console, to a rolling file, and — where it is configured — to Azure Monitor / Log Analytics or the local Aspire dashboard over OpenTelemetry.

For an investigation you can raise the level for a while — everything to Debug, or one area to Trace — without restarting anything. The change ends by itself at the time you give, and it can never silence audit events or the exception log.

How a level is decided

flowchart LR
    Baseline["Logging:LogLevel<br/>(appsettings, per service)"]
    Runtime["Logging:Runtime<br/>default + overrides + expiry"]
    Pinned["Pinned: CommandCenter.Audit,<br/>the exception log (Information)"]
    Controller["LogLevelController"]
    MEL["Filter rules<br/>(every provider)"]
    Serilog["Serilog switches<br/>(console, file)"]
    OTel["OpenTelemetry<br/>(Azure Monitor, Aspire)"]

    Baseline --> Controller
    Runtime --> Controller
    Pinned --> Controller
    Controller --> MEL
    Controller --> Serilog
    MEL --> Serilog
    MEL --> OTel

Hold "Alt" / "Option" to enable pan & zoom

• The baseline is each service's Logging:LogLevel (usually Default: Information, Microsoft.AspNetCore: Warning).
• The runtime levels (Logging:Runtime) add a new default and per-category overrides, only until ExpiresAt. The most specific category wins, so a new default of Debug does not make Microsoft.AspNetCore chatty; give it an override if you want that too.
• Pinned categories always record Information and above: audit events (CommandCenter.Audit) and the exception log entries the Log Analytics Exceptions page reads. No runtime change and no provider-specific setting can lower them.
• One controller turns all of this into the filter rules every provider uses, and moves Serilog's level switches to match, so the console, the file and Azure Monitor always agree.

Setting a runtime level

Until the Logging page exists (/config/logging), set the section in the service's configuration (App Configuration once it is wired; appsettings.json reloads too):

"Logging": {
  "Runtime": {
    "Default": "Debug",
    "Overrides": { "CommandCenter.WebApi.Agents": "Trace" },
    "ExpiresAt": "2026-09-28T16:00:00Z",
    "Reason": "ticket 1234: agent heartbeats",
    "SetBy": "ruben"
  }
}

• ExpiresAt is required. Without it, or once it has passed, nothing changes — a raised level can never be left on by accident. When it passes, the baseline applies again on its own, within a second, with no configuration change.
• Levels: Trace, Debug, Information, Warning, Error, Critical, None. An override names a category prefix (a namespace or a class).
• Applying and ending are themselves audit events (EventIds 900 and 901): who, why, until when.

sequenceDiagram
    participant Operator
    participant Config as Configuration
    participant Service
    Operator->>Config: Logging:Runtime (Debug until 16:00, reason)
    Config-->>Service: reload
    Service->>Service: rules + switches changed
    Service-->>Operator: audit 900 "Runtime log levels applied … until 16:00"
    Note over Service: 16:00
    Service->>Service: baseline again
    Service-->>Operator: audit 901 "Runtime log levels ended"

Hold "Alt" / "Option" to enable pan & zoom

Audit events

An audit event says who changed, deleted or switched what. They are written with IAuditLogger (category CommandCenter.Audit) and keep their service's EventIds, so existing queries still match:

EventId	Event
900, 901	Runtime log levels applied / ended
3081	A superseded deployment database dropped by the cleanup job
3111	A customer stage deleted
3113	A stage database dropped by a background task
3116, 3118	Customer data exported / imported
3132	A customer stage deactivated
3150	A server feature switched
3170–3176	Agents accepted or rejected; servers created or updated; maintenance set, ended or removed

To find them in Log Analytics: AppTraces | where Properties.CategoryName == "CommandCenter.Audit".

For the developer

• Log an audit event through IAuditLogger (from DI) with a [LoggerMessage] method taking this IAuditLogger logger, in your service's EventId range. Use it for changes a person makes or destructive work a task does, never for routine activity.
• Tests pass NullAuditLogger.Instance.
• Do not add a Serilog:MinimumLevel section: levels come from Logging, and a Serilog minimum would be overridden by the switches anyway.