Systems

What I Build

Open-source packages anyone can install or read, and the private production systems beside them. Each is described by what it does when something goes wrong.

Open Source

The Packages

Each one installs from PyPI, and each opens to what it does and how it is kept honest.

Python

dagster-dex

Let dex read your Dagster asset graph the way it reads a dbt project: keys, joins, sources, semantics, drift.

pip install dagster-dex
What it finds, and how it is kept honest

dex reports where a warehouse has drifted from the structure a project declares. dagster-dex makes a Dagster asset graph readable the same way, without the graph having to pretend to be a dbt project first.

What It Finds

Its README walks one example end to end: a small Dagster asset graph in front of a local DuckDB warehouse, broken by hand, with no cloud account and no credential. dex finds a duplicate key and names the downstream model it impacts, which only the asset graph connects. It finds a dropped source table and names the file that declared it.

When a measure is redefined, it reports definition_changed, which no inspection of the warehouse can find, because nothing in the warehouse changed. And it says the new expression no longer maps to a column it can check, rather than passing it. When it cannot check something, it says so.

How It Is Kept Honest

  • The README is tested. CI runs the example on every pull request and refuses the change if any output line the README quotes is not one the run printed.
  • Released and licensed. Published to PyPI under Apache-2.0, with a changelog, a security policy and contribution guidelines.
Python · Alpha

ration

A ration is a bounded claim on a finite shared pool, granted for a period, and refused rather than overdrawn.

pip install catincloud-ration
How it works, and how it is kept honest

Underneath is one coordination primitive: an atomic compare-and-set against a monotonically increasing version, on a single key. A lease, a pool of leases, an address (where is the process that holds this role?) and a publication pointer are all built on exactly that primitive, and none adds an operation of its own.

The distribution is catincloud-ration and it imports as ration. The bare name on PyPI belongs to an unrelated project.

How It Is Kept Honest

  • The suite is the contract. A conformance suite decides whether a storage backend qualifies. It is run against stores built wrong on purpose, one per property, and it must reject each of them.
  • It has already found what green runs missed. Building the pool showed that a retrying holder takes a second slot unless it looks for its own grant first, and that the obvious test for this cannot fail. Only a deliberately wrong implementation revealed it.
  • In use. The estate's own dex service builds ration into its image, from this public repository, and uses its lease to announce which process holds the service's role.

It is alpha. Its README asks readers to treat every statement as intent unless it points at a passing test.

Production · Private

Catalyst Engine

Catalyst is the estate's data engine. Every day it reads a set of APIs, transforms what they return, and writes the result to a warehouse in Google Cloud, where reporting tools and the dex service read it. It runs the estate's own telemetry and a client's marketing reporting, unattended. Its source is private, so it is described here by how it works.

The warehouse lives apart from the host that computes it, so the host can be destroyed and rebuilt from code without touching the data. A deploy runs only from a release the owner cuts, and each one is smoke-tested.

Architecture

From API to Warehouse

Catalyst sits between the APIs it reads and the tools that query its warehouse.

  1. Sources

    Read daily from each API or export

    • Google Ads, GA4 and Search Console
    • Cloudflare: edge traffic and Web Analytics
    • GitHub: stars and forks of the public repositories it follows
    • GCP billing, from its BigQuery export, and Hetzner servers

    A failed call is retried with backoff

  2. Catalyst Engine

    Compute, kept apart from the data

    • Dagster runs each day's partition on a schedule
    • Polars transforms it in memory
    • dbt builds the client's reporting marts
    • Asset checks test data contracts and freshness
    • Its models and their semantics are published as one project, behind a ration pointer
    • Docker on Hetzner, with LUKS full-disk encryption, reached over Tailscale

    Parquet files and BigQuery tables

  3. Storage

    Google Cloud

    • Cloud Storage: the raw and staging Parquet, and the published project
    • BigQuery: staging tables and the reporting marts

    SQL, and the published project

  4. Queried By

    Over SQL, or through the published project

    • BI tools, over SQL
    • The dex service, which reads the published project and answers an agent's questions of the warehouse under a budget
How It Fails

What Green Missed

Each card is a failure that reached production before a control existed to catch it, and what Catalyst does about it now.

A Schedule Stops Silently

After the host was rebuilt, the pipeline's schedules came back switched off, and the reporting tables served stale data while every check stayed green: a check on data that never arrives never runs. Every schedule's state is now declared in code and compared with the live one, and freshness deadlines raise an alert when a source's data does not arrive in time.

A Number Is Wrong, and Every Check Passes

A cost table reported zero for one provider on days whose source data held real spend. Its row count, schema and keys were all correct, so nothing failed. Catalyst now recomputes a reporting table's values from the data it was derived from and compares the two, on a schedule of its own, because rebuilding the table is the very act that would hide the gap.

The Warehouse Has Holes

Days sat missing from the middle of one table for weeks, and the newest day of another held rows with no match in the table they join to. Each table was fresh and present. Catalyst now checks that dates and keys run without gaps, and that every declared join holds in the warehouse itself.

A Run Is Lost or Stuck

A deploy that landed while a job ran killed its worker, and nothing marked the job: such jobs once held the queue shut for every schedule. Another job waited for hours behind a slot a cancelled run still held. A deploy now waits, for a bounded time, while a job runs. Catalyst cancels a job whose worker no longer exists, frees a stopped run's slots as soon as it stops, and raises an alert when a job has waited far longer than the longest wait seen on the host.

An alert from any of these is posted to the estate's operations channel.

Production · Private

The dex Service

dex is an open-source tool that lets an agent explore a warehouse, build models on it and watch it for drift. The estate runs it as a service of its own, over the warehouse Catalyst builds, so an agent can ask the warehouse a question without being handed the warehouse. Its source is private, so it is described here by how it works.

The service is a checkpoint. Its callers include scheduled workflows and an agent that answers questions in Slack, and neither holds a warehouse credential. The callers decide when and why a query runs. The service decides whether it may run, and how much it may cost.

How It Behaves

Checked, Bounded & Refused

Each part is described by what it does when a request should not go through.

Every Query Is Checked First

A question arrives as one read-only SELECT with a budget. It passes a SQL firewall, a PII policy and a byte-budget check before the warehouse runs it. A refused query comes back in the same envelope as an answer, with a status the caller can branch on without parsing prose.

It Fails Closed

With no credential configured, every guarded route answers 503 rather than serving unprotected. Spend is metered in one ledger shared across hosts, so hosts cannot each hold their own ceiling against one bill. When the ledger cannot be reached, a billed call is refused and routes that never bill keep serving.

A Fix Is a Plan First

Beyond questions, the service reports what has changed in the warehouse since a pinned baseline, and proposes a fix as a stored plan that can be reviewed. Proposing writes nothing to the project, and the agent cannot apply a plan: applying is a separate step that a person approves.

One Active Host at a Time

More than one host runs the service, and one of them is the active one at a time, announced through a ration lease. A caller looks up which host is active and sends the lease version it saw with each request. Such a request is refused by a host that is not active, and when its version predates the last handover, so a stale address is not answered by the wrong host. If the lease store is down, the active host keeps answering.

What running dex in production turns up goes back upstream, as issues and fixes in the public repository.

Behind the Systems

How They Were Built

The log records the decisions behind them: the architecture, the cost fixes and what broke along the way.