Skip to main content

Pulse Lakehouse

Operational databases are built for the transaction you're doing right now — a fast lookup, a single-row update. They're the wrong place to keep years of event history or millions of model outputs: expensive per gigabyte, and slow to scan when you finally want to ask a question across all of it. Lakehouse is the other half of that story. It's where analytical data goes to be kept cheaply and queried later — the sales history behind a forecast, the log of every decision an agent made, a dataset you exported to reason over.

The catch with most cheap-storage answers is that they turn into a swamp: a bucket of files nobody can query without remembering how each one was laid out. Lakehouse keeps the cheapness without the swamp. Every dataset declares its shape up front, so it stays a real, queryable table — while still being ordinary columnar files underneath, priced like storage rather than like a database.

Creating a dataset​

A dataset is a table with a declared schema. You say what columns it has and how it's partitioned, and from then on it enforces that contract:

POST /@<projectSlug>/analytics/<slug>
{
"displayName": "Q2 Sales Data",
"residencyClass": "shared",
"columns": [
{ "name": "order_id", "type": "string" },
{ "name": "amount", "type": "decimal(12,2)" },
{ "name": "sku", "type": "string" },
{ "name": "occurred_at", "type": "timestamp" }
],
"partitionBy": ["occurred_at:day"]
}

residencyClass is required: per-subject or shared. Column types are string, int32, int64, float64, bool, date, timestamp, json, bytes, and decimal(p,s). A few choices here save you pain later. Partitioning by a time dimension (occurred_at:day) means a query for last week's data reads last week's files and skips the rest — the difference between a fast answer and a full scan. Money is decimal, never float, so totals stay exact. And the region your project is pinned to is prepended to every dataset's partitioning automatically, so analytical data never drifts out of its jurisdiction.

Write recordsnewline-delimited JSONUpload a fileCSV or Parquet fileMaterialize a collectionfrom Governed DatabaseDatasetdeclared-schema tableappend-onlyadditive schema changespartitioned by timeregion pin prependedto the partitioningPlatform-managed storagethe defaultYour object storagechosen per datasetS3, GCS, Azure Blob,R2, B2, S3-compatiblecredential held by Kiboquery: dataset.<name>
Three ways in, one governed dataset, storage you can point at your own account, and a single query call to read it back.

Datasets are append-only and their schemas evolve additively: you can add a column, but you can't silently change what an existing one means, and you don't update individual rows in place. That's deliberate — analytical history you can quietly rewrite isn't history you can trust a forecast or an audit to.

Getting data in​

Three ways, depending on where your data is coming from:

  • Write records directly — POST /@<projectSlug>/analytics/<slug>/write takes newline-delimited JSON, for a service or an agent streaming records in as they happen.
  • Upload a file — POST /@<projectSlug>/analytics/<slug>/upload walks a CSV or Parquet file through infer → review → commit: the platform reads the file, proposes a schema, and lets you adjust it before anything lands.
  • Materialize a collection — turn an operational Governed Database collection into an analytical dataset without an export-and-reimport round-trip, when you want a queryable snapshot of live data to analyze against.

A dataset is read back with the same query call you use for raw SQL, by qualifying the table as dataset.<name>:

Create the client once

client
import { createClient, realtimeTopic } from "@k9o/core";

const k9o = createClient({
  orgSlug: "acme",
  projectSlug: "task-app",
  apiKey: process.env.K9O_API_KEY,
});

Pulse · Lakehouse

Pulse Datasets are columnar Parquet, queried with the same k9o.query by qualifying the table as dataset.<name>. They are read-only. The platform's own telemetry lands in dataset.telemetry_metrics, dataset.telemetry_logs and dataset.telemetry_traces.

1. Hourly latency
// Pulse Datasets are queried with the same k9o.query, by qualifying the table as dataset.<name>.
// Literal values go in the text of the query.
const { data: hours } = await k9o.query<{
  hour: string;
  samples: number;
  avg_ms: number;
  p95_ms: number;
}>(
  `SELECT date_trunc('hour', occurred_at) AS hour,
          count(*)                          AS samples,
          avg(value)                        AS avg_ms,
          quantile_cont(value, 0.95)        AS p95_ms
     FROM dataset.telemetry_metrics
    WHERE name = 'http.latency' AND occurred_at > now() - INTERVAL 7 DAY
    GROUP BY hour
    ORDER BY hour DESC`,
);

// Use the rows: find the slowest hour of the week.
const worst = hours.reduce((a, b) => (b.p95_ms > a.p95_ms ? b : a));
console.log(`${hours.length} hours of data`);
console.log(`slowest hour: ${worst.hour}, p95 ${Math.round(worst.p95_ms)} ms over ${worst.samples} samples`);
Output · bash
168 hours of data
slowest hour: 2026-10-29T14:00:00.000000Z, p95 912 ms over 4210 samples
  • Bind parameters (:name, $1) work on a dataset query as well; this example writes its value into the text of the query.
  • A query that mentions a dataset runs on the analytics engine; one that names only collections runs on the primary database. The SDK can create and feed a dataset (k9o.analytics.create and k9o.analytics.write); the examples here only read the datasets.

Keeping data in your own account​

By default a dataset lives on platform-managed storage. But you can point it at your own object storage instead — S3, GCS, Azure Blob, Cloudflare R2, Backblaze B2, or any S3-compatible endpoint — chosen per dataset when you create it. The credential for that binding is held by Kibo, under the same custody model as every other secret on the platform.

This is the sovereignty claim a shared warehouse structurally can't make: your analytical data physically sits in your storage account, in your jurisdiction, and Kilimanjaro queries it there. You're not trusting a promise about where the data is — you can see the bucket.

GCS — required IAM role (service account self-impersonation)​

If you're binding to Google Cloud Storage, the service account whose key you register must be able to create access tokens for itself: grant it roles/iam.serviceAccountTokenCreator on the service account itself.

gcloud iam service-accounts add-iam-policy-binding SA_EMAIL \
--member="serviceAccount:SA_EMAIL" \
--role="roles/iam.serviceAccountTokenCreator"

Replace SA_EMAIL with the service account's email — the same address you register on the binding.

Kilimanjaro's managed mint uses IAM generateAccessToken to bound every issued read credential to a short lifetime. Without this self-role, the mint fails closed and analytics reads over your bucket won't work — if GCS BYOS reads are failing, this is the first thing to check.

See also​

  • Compute — query a dataset, and join it against your live operational data.
  • Overview — how Lakehouse fits the rest of Pulse.
  • Kibo — how bring-your-own-storage credentials are held.