ShippedOct 2026SWE pipelineData infrastructureQuantBackend

TickVault

Coinbase and Kraken L2 order-book capture and replay into byte-identical, DuckDB-queryable Parquet.

  • Python
  • DuckDB
  • Parquet
  • WebSockets
  • Dagster
  • Streamlit
FIG_001[ L2 BOOK REPLAY, SIMULATED — HOVER TO PROBE ]

Results at a glance

Replay throughput
9.6–10.2k msg/s
three runs of tickvault bench replaying recorded sessions
source for Replay throughput
Peak memory
223 MB
peak RSS during the replay benchmark
source for Peak memory
Unexplained anomalies
0
in a 30-minute live capture of 476,965 messages; 3 reconnects, each closed by a resync
source for Unexplained anomalies
Tests
448
offline suite, run in CI with lint and strict type checks
source for Tests

The problem

Coinbase and Kraken publish level-2 order books as one snapshot followed by a stream of incremental updates over a websocket. Lose a single update and every book you build from that stream afterwards is wrong, usually without any visible error. Anything downstream, from a spread chart to a backtest, inherits the mistake. I wanted a capture pipeline where that failure is impossible to miss: every gap is detected and repaired, every anomaly is either explained or fails the quality check, and the stored tables can be rebuilt exactly from what the venues sent.

What I built

TickVault streams L2 books and trades for BTC-USD and ETH-USD from Coinbase Advanced Trade and Kraken WS v2. It writes every websocket frame to a raw log before parsing it, normalizes the messages into one model, applies them to a book per venue and symbol, and writes hourly partitioned Parquet tables for trades, book deltas, book snapshots and events. Prices and sizes are stored as exact decimals and timestamps as nanoseconds. DuckDB views sit over the tree, and one CLI covers ingest, replay, compaction with SHA-256 manifests, QA, reconciliation against Tardis.dev trades and a replay benchmark. A Streamlit dashboard serves a bundled 30-minute sample, and a static QA report is published on GitHub Pages.

Architecture

FIG_002[ CAPTURE → VALIDATE → STORE — HOVER A STAGE ]

The live path is connectors, then the raw writer, then pure normalizers, then the book processor, then the Parquet writer. The processor keeps each symbol in one of two states, SYNCING or LIVE. When it sees a sequence gap, a checksum mismatch or a broken invariant, it writes an events row, moves the symbol to SYNCING and sends a resync request back to the connector. Deltas that arrive while a symbol is SYNCING are dropped and counted, and the count is recorded on the resync event once a fresh snapshot makes the book LIVE again. Replay reads the raw files in a fixed order and pushes them through the same normalizer, processor and writer, so there is only one code path to trust. Downstream, tickvault qa aggregates the tables per venue, symbol and day and exits 1 if any anomaly has no later resync.

The hard part

The hard part was getting the protocol edges exactly right, and each venue had its own.

Coinbase: sequence numbers per connection

I checked the venues live before writing the connectors. On the Coinbase Exchange feed, the L2 channels that still work without auth carry no sequence numbers, so I moved to the Advanced Trade feed, where every message carries a sequence_num counter scoped to its connection and shared by all channels on it. TickVault opens one connection per product, so a gap only affects one product. The tracker expects last + 1, and anything higher is one gap event. The message that revealed the gap is discarded and the symbol goes to SYNCING. Resync unsubscribes and resubscribes level2 on the same socket, which I verified live returns a fresh full-depth snapshot in about half a second. After a reconnect, deltas from the older connection are always dropped, so a stale update can never land on a book that was rebuilt from a newer snapshot.

Kraken: CRC32 over the exact digits

Kraken v2 has no sequence numbers. Instead, every book message carries a CRC32 of the top 10 asks and bids, computed from the price and quantity text with the decimal point removed and leading zeros stripped. Kraken sends those values as JSON numbers, so a float parse turns 0.00006120 into 6.12e-05, drops the trailing zero and breaks every checksum. The normalizer decodes numbers straight to Decimal and rejects floats outright, and the checksum code formats with fixed-point notation rather than str(), which can switch to exponent form for small values. On a live BTC/USD snapshot the raw digits produced the venue’s checksum and the float path did not.

Byte-identical replay on a write-once log

Raw files are created with exclusive create, so no code path can reopen or overwrite one. Each line stores the frame as received, with its receive time, connection id and ingest sequence, and the writer flushes a zstd block at least once a second, so a crash loses about a second of data and a torn tail is counted, never raised. For replay to be byte-identical, nothing in the output may depend on wall-clock time: lines are ordered by receive time, connection and ingest sequence, Parquet buffers flush on a logical clock taken from the records, event details are JSON with sorted keys, and file names come from the first row’s connection id and ingest sequence.

How I proved it works

The suite has 448 tests and runs offline; checks that talk to the live venues are marked and opt-in. CI runs ruff, strict mypy and pytest as three required jobs. Tests cover a single missing sequence number being flagged and resynced, a pre-reconnect delta never being applied after a resync, the checksum against reference vectors taken from the live feed, crossed books and other invariant violations, two replays producing byte-identical Parquet, and raw files hashing the same before and after every command. A deterministic synthetic session with one Coinbase gap and one Kraken checksum failure is checked against an exact expected event list, and property tests check the sequence tracker for false positives.

Then I ran it live for 30 minutes on 2026-10-15. It captured 476,965 raw messages across both venues and wrote 26,294 trades and 1,840,670 book deltas. The run had 3 reconnects and 3 resyncs, with 0 gaps, 0 checksum failures, 0 invariant violations and nothing unexplained.

Results

On a MacBook Pro with an Apple M5 and 32 GB, three runs of tickvault bench replayed recorded sessions at 9.6 to 10.2 thousand messages per second overall, with peak RSS of 223 MB. The live rate across both venues was about 265 messages per second, so replay has plenty of headroom; the venues set the live rate. The one ugly number from the capture is a Coinbase BTC-USD p99 receive lag of about 23 seconds, which came from the snapshot burst after one reconnect, not from normal delivery. The dashboard, the QA report and the source are public, and each figure on this page links to the line in the repository it comes from.

What I’d do differently

I would build the streaming raw reader first. The first version sorted every raw line in memory, which was fine at fixture scale but made memory grow with the size of the raw tree, so I later replaced it with a lazy merge across files and an external sort for files that arrive out of order. I would also pin every chart to UTC from the start. The hosted dashboard first formatted timestamps in the viewer’s local time while the axis titles said UTC, and I had to rebuild the charts and re-record the media before the release. On scope, reconciliation is only fixture-tested against Coinbase trades so far. The roadmap’s next steps are a features table with a lookahead test, a third venue, and later always-on ingestion with a public QA history.

How I used AI

I planned TickVault as a sequence of epics and built it in waves of parallel workstreams, with AI agents writing code inside contracts I defined first. That is why Epic 0 exists: it fixed the shared models, interfaces, fixtures and architecture document before any feature work, so no workstream ever had to edit a shared file, and only one agent per wave was allowed to change the lockfile. I kept the evidence for myself. The venue behaviour the code relies on was checked against the live feeds before it went into the design, the Kraken checksum was confirmed against a live snapshot, and every figure in the README comes from pasted command output. The tests, strict type checks and byte-identical replay checks were the gate every change had to pass, whoever wrote it.

Screenshot of the TickVault dashboard titled "TickVault — crypto market-data quality". A sidebar filters venues (coinbase, kraken) and symbols (BTC-USD, ETH-USD). A row of tiles reads Messages/s (last 5 min) 158.0, Gaps 0, Checksum failures 0, Reconnects 0 and Unexplained 0, above a Message throughput line chart with one line for Coinbase and a higher one for Kraken.
FIG_003[ SCREENSHOT ]