Skip to main content

Observability

Observability on Kilimanjaro is structured around the audit log — every API call emits an event with the calling identity, the resource touched, and the outcome, recorded before the response returns to the caller. The audit log is the canonical observability surface; per-execution function logs and customer-side dashboards compose against it.

1. The audit log as the canonical surface​

Every audit event shares a common envelope per API → Security §2.1:

FieldWhat it carries
eventIdevt_<base62> — unique per event
timestampISO 8601 UTC at sub-millisecond precision
eventTypeDot-separated hierarchy (auth.session.created, data.write.documents, etc.)
projectId / orgIdThe project + org the event belongs to
actorId / actorTypeWho did the thing (USER / SERVICE_ACCOUNT / PLATFORM)
actorRolesPlatform roles the actor held at event time
ipAddress / userAgentCaller context
requestIdTrace identifier linking related events
resultSUCCESS or FAILURE
contextEvent-specific payload; schema varies by eventType

The audit envelope is the same for human-driven sign-ins, service-token operations, extension hook invocations, and platform-internal events. The eventType family is a closed enumeration per the audit catalogue — auth.*, data.write.auth.*, data.*, storage.*, security.*, lifecycle.*, extension.*, data.write.approval.*.

2. Forensic attribution​

The platform's audit surface differs from a generic application audit log in one structural property: forensic attribution at the kernel. Every end-user identity event (data.write.auth.* family attached to Gate flows) captures the actual identity of both the actor and the affected end-user, with the identity established by Gate's cryptographic session before the response returns.

PII column values are excluded from the audit envelope — field names appear, field values do not. This preserves audit-channel PII-tightness (the audit log itself isn't a PII reservoir) without losing forensic depth (the per-event identity binding is platform-signed at the kernel rather than reconstructed from application logs).

For investigations, this means: reconstructing who did what to whom is a single audit-log query, not a multi-system correlation exercise.

3. Retention tiers​

Audit events live in two tiers per API → Security §2.3:

TierWindowStorageAccess
Hot90 days (default)Postgres audit-log table in the project's data-plane regionQueryable via the audit-log query API
ColdUp to 7 years (org-level setting)Long-term archival bucket in the same jurisdictionRetrievable only via export

After the hot-tier TTL, events tier to cold storage automatically. The retention setting is org-level; project-specific overrides are not modelled.

API callevery requestAudit eventrecorded before theresponse returnsHot tier: 90 days (default)Postgres table in the data-plane regionqueryable via the audit-log query APIafter the hot-tier TTLCold tier: up to 7 yearsarchival bucket, same jurisdictionretrievable only via exportSIEM webhookHMAC-signed POST within 5 seconds
One audit event feeds the hot tier, which tiers to cold storage, and the SIEM webhook.

4. Querying the audit log​

The query API supports cursor-paginated lookup filtered by event type (a prefix match on the stored event type), actor identity, and time range:

GET <orgSlug>.api.k9o.io/@<projectSlug>/audit-log?eventType=data.write.approval&actorId=<actorId>&after=2026-05-01T00:00:00Z&limit=100

Auth floor: PROJECT_ADMIN or PROJECT_AUDITOR. The PROJECT_AUDITOR role exists specifically for compliance reviewers who need read access to the audit log without write authority on project data — a separate-dimension role cut that doesn't compose into the project-ladder hierarchy.

Window-bounded queries are limited to 90 days (hot-tier TTL); larger windows require export.

5. Audit export​

For windows beyond hot tier or for offline retention:

POST <orgSlug>.api.k9o.io/@<projectSlug>/exit/export
{ "scope": { "auditLog": true } }

Async — returns 202 Accepted with { id, status, deliveryMethod }. Poll GET <orgSlug>.api.k9o.io/@<projectSlug>/exit/export/<jobId> until status is completed; the status carries an expiresAt, and the export is then fetched from GET .../exit/export/<jobId>/download. A data.export.completed audit event is emitted on completion, so the export itself appears in the audit log.

Only one export job runs per project at a time; concurrent requests return AUDIT_LOG_EXPORT_IN_PROGRESS (409).

6. SIEM forwarding​

For streaming events to your external observability surface, configure a webhook destination per API → Security §2.6:

POST <orgSlug>.api.k9o.io/@<projectSlug>/security/siem
{
"enabled": true,
"webhookUrl": "https://siem.example.com/k9o-events",
"hmacSecretRef": "<vault secret reference>",
"eventTypes": ["security.*", "auth.*"],
"format": "json"
}

Delivery mechanics:

  • Events delivered via HTTP POST to webhookUrl within 5 seconds of emission.
  • X-K9o-Signature header carries sha256=<HMAC-SHA256(body, secret)>, computed with the secret that hmacSecretRef points to — verify before processing.
  • At-least-once delivery with exponential-backoff retry (initial 30s, max 15min, for up to 48h).
  • After 48h of delivery failure, forwarding pauses + a security.siem.delivery_failed audit event fires; forwarding resumes automatically once the endpoint is reachable.

The SIEM-forwarding contract is identical for any external observability surface — Splunk, Datadog, Elastic, OpenSearch, customer-internal endpoints. SIEM integration is included on every Kilimanjaro tier (not a premium-tier feature).

A test endpoint takes no body and runs a dry run: it reports the configured webhookUrl and format and the sample event it would send. It answers 409 if no webhook is configured yet.

POST <orgSlug>.api.k9o.io/@<projectSlug>/security/siem/test

7. Function execution logs​

Function invocations emit a function.invocation audit event; see API → Compute.

Log capture is per-invocation:

  • Stdout / stderr from the function process is captured to the platform's log store.
  • Logs are retained for 28 days in the platform's telemetry sink (the Pulse OLAP retention window); export beyond that is available via the durable Elastic Analytics replay path.
  • Real-time streaming via Server-Sent Events for the currently-executing invocation.
  • Bulk retrieval via the function-logs query API for completed invocations within the retention window.

For long-running agentic workloads, the streaming SSE surface gives you real-time observability during execution without waiting for the function to complete.

8. Customer-side observability surfaces​

The audit log is the canonical platform-side observability surface. For customer-side surfaces — dashboards, charts, internal operational tooling — the Lookout extension composes against project collections and the audit log itself.

Common patterns:

  • Audit-log dashboards — Lookout reads from the _sys.audit_log collection (where the audit events live) to build operational dashboards. Each dashboard inherits the same JSON-policy filter that governs audit-log read access, so a PROJECT_AUDITOR viewing the dashboard sees the same scoped event subset.
  • Error-rate-by-endpoint — a Lookout chart bound to a saved query against the audit log, filtered to result: FAILURE, grouped by eventType, time-binned for trend visualisation.
  • End-user activity — for projects whose Gate-issued identities authenticate their own application's customers, a dashboard against the data.write.auth.* event family shows end-user sign-in patterns by region, by IdP, by MFA-completion status.

9. Signed audit chain​

Hot-tier and cold-tier exports carry a cryptographic chain — each event's eventId references the prior event's hash, forming a tamper-evident sequence. This is the audit-log's structural integrity guarantee: an event cannot be silently inserted, removed, or modified without breaking the chain.

The chain is anchored against an external transparency log (configurable per project) for cross-platform verification — the customer can independently verify the platform-side audit log against an out-of-band reference. Third-party attestation of the chain is the binding for the sovereignty claim per Guides → Sovereignty.

10. See also​