Observability
Observability on Kilimanjaro is structured around the audit log — every API call emits an event with the calling identity, the resource touched, and the outcome, recorded before the response returns to the caller. The audit log is the canonical observability surface; per-execution function logs and customer-side dashboards compose against it.
1. The audit log as the canonical surface
Every audit event shares a common envelope per API → Security §2.1:
| Field | What it carries |
|---|---|
eventId | evt_<base62> — unique per event |
timestamp | ISO 8601 UTC at sub-millisecond precision |
eventType | Dot-separated hierarchy (auth.session.created, data.write.documents, etc.) |
projectId / orgId | The project + org the event belongs to |
actorId / actorType | Who did the thing (USER / SERVICE_ACCOUNT / PLATFORM) |
actorRoles | Platform roles the actor held at event time |
ipAddress / userAgent | Caller context |
requestId | Trace identifier linking related events |
result | SUCCESS or FAILURE |
context | Event-specific payload; schema varies by eventType |
The audit envelope is the same for human-driven sign-ins, service-token operations, extension hook invocations, and platform-internal events. The eventType family is a closed enumeration per the audit catalogue — auth.*, data.write.auth.*, data.*, storage.*, security.*, lifecycle.*, extension.*, data.write.approval.*.
2. Forensic attribution
The platform's audit surface differs from a generic application audit log in one structural property: forensic attribution at the kernel. Every end-user identity event (data.write.auth.* family attached to Gate flows) captures the actual identity of both the actor and the affected end-user, with the identity established by Gate's cryptographic session before the response returns.
PII column values are excluded from the audit envelope — field names appear, field values do not. This preserves audit-channel PII-tightness (the audit log itself isn't a PII reservoir) without losing forensic depth (the per-event identity binding is platform-signed at the kernel rather than reconstructed from application logs).
For investigations, this means: reconstructing who did what to whom is a single audit-log query, not a multi-system correlation exercise.
3. Retention tiers
Audit events live in two tiers per API → Security §2.3:
| Tier | Window | Storage | Access |
|---|---|---|---|
| Hot | 90 days (default) | Postgres audit-log table in the project's data-plane region | Queryable via the audit-log query API |
| Cold | Up to 7 years (org-level setting) | Long-term archival bucket in the same jurisdiction | Retrievable only via export |
After the hot-tier TTL, events tier to cold storage automatically. The retention setting is org-level; project-specific overrides are not modelled.
4. Querying the audit log
The query API supports cursor-paginated lookup filtered by event type (a prefix match on the stored event type), actor identity, and time range:
GET <orgSlug>.api.k9o.io/@<projectSlug>/audit-log?eventType=data.write.approval&actorId=<actorId>&after=2026-05-01T00:00:00Z&limit=100
Auth floor: PROJECT_ADMIN or PROJECT_AUDITOR. The PROJECT_AUDITOR role exists specifically for compliance reviewers who need read access to the audit log without write authority on project data — a separate-dimension role cut that doesn't compose into the project-ladder hierarchy.
Window-bounded queries are limited to 90 days (hot-tier TTL); larger windows require export.
5. Audit export
For windows beyond hot tier or for offline retention:
POST <orgSlug>.api.k9o.io/@<projectSlug>/exit/export
{ "scope": { "auditLog": true } }
Async — returns 202 Accepted with { id, status, deliveryMethod }. Poll GET <orgSlug>.api.k9o.io/@<projectSlug>/exit/export/<jobId> until status is completed; the status carries an expiresAt, and the export is then fetched from GET .../exit/export/<jobId>/download. A data.export.completed audit event is emitted on completion, so the export itself appears in the audit log.
Only one export job runs per project at a time; concurrent requests return AUDIT_LOG_EXPORT_IN_PROGRESS (409).
6. SIEM forwarding
For streaming events to your external observability surface, configure a webhook destination per API → Security §2.6:
POST <orgSlug>.api.k9o.io/@<projectSlug>/security/siem
{
"enabled": true,
"webhookUrl": "https://siem.example.com/k9o-events",
"hmacSecretRef": "<vault secret reference>",
"eventTypes": ["security.*", "auth.*"],
"format": "json"
}
Delivery mechanics:
- Events delivered via HTTP POST to
webhookUrlwithin 5 seconds of emission. X-K9o-Signatureheader carriessha256=<HMAC-SHA256(body, secret)>, computed with the secret thathmacSecretRefpoints to — verify before processing.- At-least-once delivery with exponential-backoff retry (initial 30s, max 15min, for up to 48h).
- After 48h of delivery failure, forwarding pauses + a
security.siem.delivery_failedaudit event fires; forwarding resumes automatically once the endpoint is reachable.
The SIEM-forwarding contract is identical for any external observability surface — Splunk, Datadog, Elastic, OpenSearch, customer-internal endpoints. SIEM integration is included on every Kilimanjaro tier (not a premium-tier feature).
A test endpoint takes no body and runs a dry run: it reports the configured webhookUrl and format and the sample event it would send. It answers 409 if no webhook is configured yet.
POST <orgSlug>.api.k9o.io/@<projectSlug>/security/siem/test
7. Function execution logs
Function invocations emit a function.invocation audit event; see API → Compute.
Log capture is per-invocation:
- Stdout / stderr from the function process is captured to the platform's log store.
- Logs are retained for 28 days in the platform's telemetry sink (the Pulse OLAP retention window); export beyond that is available via the durable Elastic Analytics replay path.
- Real-time streaming via Server-Sent Events for the currently-executing invocation.
- Bulk retrieval via the function-logs query API for completed invocations within the retention window.
For long-running agentic workloads, the streaming SSE surface gives you real-time observability during execution without waiting for the function to complete.
8. Customer-side observability surfaces
The audit log is the canonical platform-side observability surface. For customer-side surfaces — dashboards, charts, internal operational tooling — the Lookout extension composes against project collections and the audit log itself.
Common patterns:
- Audit-log dashboards — Lookout reads from the
_sys.audit_logcollection (where the audit events live) to build operational dashboards. Each dashboard inherits the same JSON-policy filter that governs audit-log read access, so aPROJECT_AUDITORviewing the dashboard sees the same scoped event subset. - Error-rate-by-endpoint — a Lookout chart bound to a saved query against the audit log, filtered to
result: FAILURE, grouped byeventType, time-binned for trend visualisation. - End-user activity — for projects whose Gate-issued identities authenticate their own application's customers, a dashboard against the
data.write.auth.*event family shows end-user sign-in patterns by region, by IdP, by MFA-completion status.
9. Signed audit chain
Hot-tier and cold-tier exports carry a cryptographic chain — each event's eventId references the prior event's hash, forming a tamper-evident sequence. This is the audit-log's structural integrity guarantee: an event cannot be silently inserted, removed, or modified without breaking the chain.
The chain is anchored against an external transparency log (configurable per project) for cross-platform verification — the customer can independently verify the platform-side audit log against an out-of-band reference. Third-party attestation of the chain is the binding for the sovereignty claim per Guides → Sovereignty.
10. See also
- API → Security — audit envelope + event catalogue + retention + query API + export + SIEM forwarding.
- API → Compute — function invocation audit event + log capture mechanics.
- Lookout — dashboarding extension for customer-side observability surfaces.
- Guides → Sovereignty — audit-log signed chain as a sovereignty verification surface.
- Architecture → Architecture — where audit emission sits in the request flow.