The Slow Half of PLAID: Making Exact Rescoring Cheap with One Int8 Operation

Every number below comes from the same setup: an Apple M4 MacBook Air with every engine running on CPU (no GPU touches the search path), idle, best-of-3 timing over full BEIR query sets (300 SciFact / 648 FiQA), with query embeddings precomputed once and fed to every engine, so no timing includes query encoding: this post measures search, not the model, on a 2×2 grid of checkpoints (LateOn, GTE-ModernColBERT) × corpora (SciFact 5k, FiQA 57k), at 4-bit residuals unless noted, with nDCG@10 scored from the bundled qrels through one shared scorer. The full protocol, including the statistical tests, is at the end.

Two stages, one bill

If you serve ColBERT-family models, you're running some descendant of PLAID, and every query pays for two stages. Stage 1 is deliberately sloppy: score the query against k-means centroids, probe the inverted lists, prune. Stage 2 cleans up after it: every shortlisted document scored completely, every query token against every document token, over the compressed codes the index stores. That full-fidelity cleanup is what I mean by “exact rescoring” throughout.

Stage 2 has always been the slow half, for an embarrassingly concrete reason: the textbook implementation decompresses. Every candidate token gets reconstructed to float32 (512 bytes at 128 dims), renormalized, and fed to a dense MaxSim. next-plaid did this too, until these two PRs.

Figure 1 is the bill, itemized. It comes from an earlier profiling session on SciFact and a 15k FiQA slice, so read it for the shape of each bar; the throughput numbers you can hold me to start at Figure 2. The top bar of each pair is 1.6.5; the bottom is the two PRs. Decompression is the single biggest line item on every top bar, and it simply does not exist on the bottom ones. It is also the roadmap for the rest of this post.

One asymmetric int8 operation

#169 fits in two equation blocks. Some notation first. Write q for one query token and t for one document token, both vectors of dimension D; MaxSim adds up the best t for each q, so everything below is one query token against one document token. The index keeps a centroid table C, one row per cluster, and the codec keeps a table W of sixteen bucket weights, the values a 4-bit code decodes to. What is actually stored for t is its centroid id cid and one code c_d per dimension; t itself is only ever implied. The exact score splits cleanly along that decomposition:

The first term is free: stage 1's query×centroid GEMM already computed it, so stage 2 just reads it back. The norm is one scalar per token, computed from the f32 weights and kept in a small sidecar file beside the index; for an index built before 1.7.0 the sidecar is built once at load, about a second per 3M tokens, which is why 1.6.x indexes work unchanged. Everything hangs on the residual term, and until recently every PLAID descendant I have read computed it the same way: decompress each stored token to float32, then run a dense dot product.

WARP (Scheerer et al., SIGIR ’25) showed the decompress step is unnecessary: its “implicit decompression” scores the codes directly through a lookup table. When I first read it I assumed skipping materialization was the whole win. It wasn’t: WARP keeps that table in f32, and when I measured it, removing the copy was necessary but not decisive. The decisive gain came from changing the arithmetic currency. So the bet in #169 is that the currency is what matters, and three things happen when the table drops from f32 to int8, all of them compounding. The table W is tiny to begin with: a 4-bit code can take sixteen values, so the codec stores sixteen bucket weights, one representative residual value per code, shared by every dimension of every token in the index. In f32 those sixteen weights are 64 bytes; in int8 they are 16, which is exactly one SIMD register, and that is what lets the gather below become a single tbl instruction instead of a four-register shuffle. A single sdot then retires 16 multiply-adds where an f32 FMA retires 4. And the working set per token drops 4×, so the kernel stays in L1 right where the f32 version falls out of it. Precision is the price, and I pay it only on the small correction term; the big centroid contribution stays f32:

One terminology note, because “asymmetric” is overloaded. I mean it in the ADC sense from the product-quantization literature: the query is never encoded with the document codebook. The stored side keeps its lossy 4-bit codes from index time while the query gets a fresh, near-lossless int8 encoding at search time, and the two sides meet without a shared quantizer.

Now the part I find genuinely pretty, and it's worth slowing down for even if you've never read a line of SIMD. In numpy terms, the residual term is two lines:

w   = W[codes]   # fancy-index gather: 128 lookups
acc = q8 @ w     # integer dot product

W[codes] is the exact same operation as embedding[input_ids]: fancy indexing. And fancy indexing is where vectorization usually goes to die, because it looks like 128 scattered memory reads per token that the CPU can't do anything clever with.

Except for one accident of sizing: the table has sixteen int8 entries, and sixteen bytes is exactly one 128-bit SIMD register. A register is the CPU's version of a tiny fixed-size numpy array, held inside the core itself and operated on whole. ARM's tbl instruction treats one register as a lookup table and a second as sixteen indices, and hands back all sixteen gathered values in a third. (This is the fast scan trick from Quick(er) ADC (André et al.), already powering ScaNN and faiss's IVFPQFastScan, applied here to PLAID's scalar residual codec instead of PQ subspaces.) The table and the indices never leave registers, and nothing is ever decoded to float. That's W[codes], sixteen dimensions at a time, in one instruction.

Here's the whole residual term for one document token on ARM, lightly simplified from the shipping kernel:

// numpy: acc = q8 @ W[codes]
//
// A NEON register is a 128-bit box: sixteen int8 lanes
// side by side. Every instruction below touches all
// sixteen lanes at once, and lane i of a result depends
// only on lane i of its inputs. 128 dims per token; each
// packed byte holds two 4-bit codes, its low and high
// halves ("nibbles").

// One 16-entry table per nibble position: W, permuted
// into the codec's bit-packing order. Two registers at
// 4 bits (8/nbits in general), loaded once per document.
tab_lo = vld1q_s8(W_lo)
tab_hi = vld1q_s8(W_hi)

// Pass 1, once per document token: expand the 64 packed
// bytes into 128 int8 weights in a stack buffer w.
for i in 0, 16, 32, 48:
  bytes = vld1q_u8(packed + i) // 16 bytes = 32 codes
  lo    = bytes & 0x0F         // lane i = low nibble
  hi    = bytes >> 4           // lane i = high nibble
  // tbl is the one lane-crossing step: lane i of the
  // result is table[lo lane i]. Sixteen lookups in one
  // instruction, and the "memory" it reads is a register.
  vst1q_s8(w + i,      vqtbl1q_s8(tab_lo, lo))
  vst1q_s8(w + 64 + i, vqtbl1q_s8(tab_hi, hi))

// Pass 2, once per query token: the dot product over
// those 128 weights, which are still hot in L1.
// sdot multiplies 16 int8 pairs lane by lane, then folds
// each group of four products into one of the four int32
// lanes. q is the query's int8 lanes, permuted once per
// search into the same plane order as w, so lane i
// always meets lane i.
for k in 0, 32, 64, 96:
  a = sdot(a, vld1q_s8(q + k),      vld1q_s8(w + k))
  b = sdot(b, vld1q_s8(q + k + 16), vld1q_s8(w + k + 16))
acc = a + b

// Per document token: 4 loads, 8 tbl, shared by every
// query token. Per query token: 8 sdot, no float. Four
// query tokens fold at once: a pairwise add puts their
// four sums in one register, then the epilogue
// (sq * sw * acc + centroid) * inv_norm runs in float lanes.

sdot deserves its own sentence: it multiplies sixteen int8 pairs and accumulates the products into int32 lanes, a sixteen-element np.dot per instruction. So the full 128-dim exact dot product for one query-token/document-token pair costs eight instructions, plus an eight-tbl expansion paid once per document token and shared by every query token. The f32 version of the same dot product needs 32 fused multiply-adds, plus the 512-byte decompressed token it operates on; this one needs a quarter of the instructions and 64 packed bytes that are already in cache. That's the 4×-per-instruction, 4×-less-traffic arithmetic behind the roofline numbers below.

If registers, lanes and tbl are new to you, I built an interactive course around this exact kernel: class 05 of the nano-plaid SIMD school steps through it with live widgets, from the first lane to the fused loop.

The asymmetry pays one more dividend. Because the weight table is query-independent, the kernel loads each document token's weights once and scores them against every query token (four at a time on NEON, eight or sixteen on x86, with the float epilogue in registers). Classic ADC builds a fresh query-dependent table for every query token before scoring; keeping the multiply inside sdot deletes that whole step. The expanded int8 weights do pass through a 128-byte stack buffer, and that is the point: written once per document token, read by every query token. Nothing is ever decoded to float, and the kernel never reads past the packed bytes.

(x86 tells the same story with different spellings: pshufb for the lookup, maddubs / AVX-512 vpdpbusd for the dot; every path is parity-tested bit-identical to the scalar reference.)

How much does int8 buy over f32? I benchmarked them like-for-like: an f32 lookup kernel with the identical loop structure runs 28.6 GMAC/s where its int8 twin runs 63.7, a 2.2× gap with everything except the arithmetic held fixed. The 4×-larger f32 working set adds a cache tax on top (my blocked-f32 prototype fell out of L1 and ran slower than its naive form). That is why WARP's f32 LUT, despite deleting decompression, barely moves the stage versus 1.6.5's decompress-then-GEMM (10.5–14.2 µs per rescored document): it changes the step but not the currency. Change the currency and the stage drops to 1.9–3.1 µs, 5–8× across the cross-platform PR benchmarks. (The production check: dropping my int8 kernel into WARP makes the same engine and pass 1.0–1.6× faster (1.3–1.5× at the probe depths where it reaches its ceiling).)

What does int8 cost in quality? Only the correction term is quantized and every norm stays exact, so the answer should be “almost nothing”, and it measures that way: ≤0.002 nDCG@10 across three checkpoints, three corpora, and bit-widths 4 down to 1; in a direct A/B on the same indexes, at most 0.0002 either direction.

Then Amdahl's law takes over: collapse stage 2, and stage 1 is suddenly most of the query. That's the second PR, #170: rework every shortlist phase with five standard algorithm swaps, all output-preserving:

In the PRs' own cross-platform benchmarks, stage 1 alone gains 4–12× depending on platform; combined, 5.6–7.1× end-to-end (the matched-quality numbers in Figure 2 are the stricter comparison). Nothing in Table 1 would make a paper, but jointly #170 is worth about as much as #169, and it only became worth doing once rescoring got out of the way. One principle covers both halves: spend precision only where the ranking needs it.

What it buys

Same machine, same embeddings, same queries, same scorer: two models, two corpora, and the original Stanford PLAID (driven as shipped, at its paper presets) as the reference point.

Three things I'd want you to take from Figure 2:

1.6.5 and PLAID are the same design, and they measure like it. 1.6.5 is PLAID's architecture in Rust, so the two are close to a tie, with 1.6.5 holding a modest edge worth about what removing the Python boundary buys. There is no quality gap hiding under the speed story either: pooled over all 1,896 paired queries, PLAID vs 1.7.0 is −0.0007, equivalent within ±0.005. The jump only appears once the architecture changes.

One index build is not enough to compare engines. Rebuilding both engines' indexes with five k-means seeds moved a single build's nDCG by ±0.003–0.008, which is as large as the difference between engines, and my first builds happened to favor PLAID's rivals. So the SciFact panels plot the mean over five builds.

PLAID was tuned, not run on defaults. Beyond its paper presets, I swept ncells × ndocs × centroid threshold jointly, 28 configurations per workload. The best of those moves PLAID's frontier by at most 15–25% at matched quality and never comes close to the 1.7.0 curve.

Bonus round: the engine that skips the stage

If exact rescoring is nearly free, the natural cross-examination is a design built on the opposite bet. WARP (SIGIR '25) folds everything into one approximate pass: lookup-table scoring over the probed cells, plus a learned estimate (t′) for every token the probe never visits:

Disclosure up front: what I benchmark is xtr-warp-rs v2.0.2, an independent Rust replication of WARP with its own adaptive controls (not the official reference implementation). It is a fairly optimized engine rather than a naive port; its author explains the design in WARP vs PLAID: inside the SOTA in multi-vector retrieval. On top of it come my patches: precomputed-embedding input, runtime arm toggles, and my int8 kernel donated into its scoring loop. I tuned it past its own auto-tuner where that helped, and I quote its best arm throughout. This isn't a horse race I staged to win; it's the cleanest test I could build of whether skipping the exact stage still buys anything once that stage is nearly free.

The comparison runs on the same 2×2 grid as Figure 2, so the frame carries over: corpus down the rows, model across the columns, throughput right, quality up. Only the cast changes. Each engine appears in both its current and its previous form, and every panel answers one question: once the exact stage is nearly free, does skipping it still buy anything? The panels disagree, and they split on corpus size rather than on model.

On 5k documents (top row), the two designs reach the same quality; next-plaid is faster on LateOn, WARP on GTE. The ceilings tie: on LateOn they differ by less than this query set can resolve (±0.013), so the race is decided on cost, and every frontier point at nDCG ≥ 0.7598 is next-plaid's. What differs is where each design lets you trade. WARP's cheap settings pay for speed in ranking quality (up to −0.029 nDCG at nprobe=4); next-plaid's cheap settings stay on the plateau. An exact second stage means the speed knob stops being a quality knob. On SciFact·GTE WARP wins outright on throughput, leading at every quality level (1,424 vs 1,325 QPS at its ceiling) with the ceilings tied within noise; at 5k with this model the one-pass design is simply the faster engine. And my kernel makes WARP 1.0–1.6× faster (1.3–1.5× at the probe depths where it reaches its ceiling) while changing its rankings not at all; the int8 arithmetic was never the point.

At 57k (bottom row), the exact stage's economics improve and the panels separate. The probe has to reach 4× further for the same quality (appendix below), and the one-pass design either cedes the frontier at every quality above nDCG 0.42 (GTE, where next-plaid is about 5× faster at WARP's best quality, and WARP's only frontier points are its two shallowest probes, well below next-plaid's cheapest setting) or hits a ceiling it cannot tune its way out of: on LateOn, WARP plateaus 0.024 nDCG below next-plaid's ceiling (p<10⁻⁴) at 4.4× the per-query cost, the grid's only quality gap that survives multiple-comparison correction, while next-plaid matches the float32-exhaustive reference. Fairness cuts both ways: WARP's auto-tuner mistuned at 57k, so I hand-swept t′ and quote its tuned ceiling, and the same sweep of next-plaid's own knobs showed the historical n_full_scores=4096 default buys nothing over 1024 here.

The takeaway

One asymmetric operation in the right place: quantize only the correction term, keep it in registers, and the exact stage of a PLAID-family engine stops being the slow half: 5–8× cheaper, on your existing indexes, at the same ranking quality. The other half of the win is stage 1: five standard techniques that only started mattering once rescoring got out of the way. The old trade of exact but slow versus fast but capped turns out, at the scales and models I tested, to have been an artifact of decompress-then-score all along. Int8 implicit decompression ships as the default in next-plaid 1.7.0 (#169 + #170); existing 1.6.x indexes load unchanged. Across the models, corpora, and scales tested here, full-quality rescoring no longer costs meaningful speed. So stop trading quality for it: run at the ceiling, and let the probe knob be a recall knob.

Citation

If this post is useful in your work, please cite it as:

@misc{hsu2026slowhalf,
  author = {Hsu, Chao-Chun},
  title = {The Slow Half of {PLAID}: Making Exact Rescoring
           Cheap with One Int8 Operation},
  year = {2026},
  month = sep,
  howpublished = {Blog post},
  url = {https://chaochunhsu.github.io/blog/slow-half-of-plaid/}
}

References

1. Keshav Santhanam, Omar Khattab, Christopher Potts, Matei Zaharia. PLAID: An Efficient Engine for Late Interaction Retrieval. CIKM 2022. arXiv:2205.09707. Reference implementation: stanford-futuredata/ColBERT.
2. Jan Luca Scheerer, Matei Zaharia, Christopher Potts, Gustavo Alonso, Omar Khattab. WARP: An Efficient Engine for Multi-Vector Retrieval. SIGIR 2025. arXiv:2501.17788. Reference implementation: jlscheerer/xtr-warp; the Rust replication benchmarked here: pau-mensa/xtr-warp-rs v2.0.2, described by its author in WARP vs PLAID: inside the SOTA in multi-vector retrieval.
3. Omar Khattab, Matei Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020. arXiv:2004.12832.
4. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, Matei Zaharia. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. NAACL 2022. arXiv:2112.01488. The residual codec next-plaid inherits.
5. Hervé Jégou, Matthijs Douze, Cordelia Schmid. Product Quantization for Nearest Neighbor Search. IEEE TPAMI 2011. The source of the asymmetric-distance (ADC) vocabulary.
6. Fabien André, Anne-Marie Kermarrec, Nicolas Le Scouarnec. Quicker ADC: Unlocking the Hidden Potential of Product Quantization with SIMD. IEEE TPAMI 2019. arXiv:1812.09162. The register-resident table lookup (“fast scan”).
7. Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar. Accelerating Large-Scale Inference with Anisotropic Vector Quantization. ICML 2020. arXiv:1908.10396 (ScaNN).
8. Matthijs Douze et al. The Faiss Library. 2024. arXiv:2401.08281 (IVFPQFastScan).
9. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, Iryna Gurevych. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS 2021 Datasets and Benchmarks. arXiv:2104.08663. Source of the SciFact and FiQA query sets and qrels.
10. next-plaid: lightonai/next-plaid. The two pull requests: #169 (int8 asymmetric rescoring) and #170 (stage-1 rework), released in 1.7.0.
11. Checkpoints: lightonai/GTE-ModernColBERT-v1 and lightonai/LateOn-regularized.