Vectors are the fundamental currency of modern AI. High-dimensional vectors — the kind that capture the meaning of a word, the semantics of a passage, or the context of a long conversation — consume vast amounts of memory, creating bottlenecks in the key-value cache that is central to every Transformer model deployed today.
TurboQuant (Zandieh et al., ICLR 2026) recently set a new state of the art for online KV cache quantization. It applies a random Haar rotation to uniformize the distribution of vector coordinates, then uses a Lloyd-Max quantizer and a QJL residual trick to achieve near-optimal distortion rates in a data-oblivious manner. Its results are impressive. But it compresses the wrong thing.
Every Transformer layer computes h = Attention(x) and then adds a residual connection: the output is x + h. The vector x is already present in the residual stream. The new information produced by attention is not h itself, but the delta delta = h - x. We show that this delta has a norm ~59% smaller than h, and that compressing delta instead of h reduces distortion by up to ×12 at 1-bit — with no additional memory cost.
The key insight: compress the delta, not the activation
TurboQuant and every prior KV cache compression method — KVQuant, KIVI, Residual VQ — compress the full attention output vector h. This is natural: h is what the model stores in the KV cache. But there is a fundamentally more efficient target available in every Transformer architecture.
The critical property, measured empirically across architectures (GPT-2, Pythia, OPT), is:
By the Panter-Dite theorem, optimal Lloyd-Max distortion scales as the square of the signal norm. Compressing delta instead of h yields:
In practice, combined with our Fisher Saturation masking (Sc = 0.769893), we measure gains of up to ×12 in distortion at 1-bit. The delta compression principle is architecture-agnostic: it applies to any model with residual connections.
How ResApexQuant works
Step 1 — ApexQuant: Fisher Saturation masking
Before compressing any vector, ResApexQuant applies ApexQuant — our base algorithm that improves over TurboQuant by exploiting the Fisher Saturation constant. After the Haar rotation y = Pi * x, we compute an adaptive scale:
where q_Sc is the empirical quantile at order Sc = 0.769893. This concentrates exactly Sc of the probability mass into [-Sc, Sc] — the "thermodynamically efficient" domain of the SR-SE framework. Coordinates outside this range are identified as outliers and receive b+1 bits of precision.
The theoretical basis: the Panter-Dite cost of Lloyd-Max quantization scales with the integral of f(x)^(1/3)dx. Restricting to [-Sc, Sc] reduces this integral by a factor proportional to Sc, delivering a provable gain over TurboQuant's unconstrained codebook.
TurboQuant SOTA
- Haar rotation
- Unconstrained Lloyd-Max
- QJL on residual
- Compresses h directly
- No skip connection exploitation
ResApexQuant OURS
- Haar rotation
- Fisher Saturation mask (Sc=0.769893)
- Outlier gate (+1 bit for |y|>σ)
- Lloyd-Max constrained in [-Sc, Sc]
- QJL on residual
- Compresses delta = h - x (b-1 bits)
Step 2 — Residual delta quantization
The second innovation is the decisive one. Instead of storing a compressed version of h, we store x (already in the residual stream, zero additional memory) plus a compressed version of delta = h - x. Because delta is geometrically smaller than h by a factor of ~0.41, we can use b-1 bits for it and still achieve dramatically lower distortion. Total memory cost is identical to quantizing h at b bits.
Key distinction from Residual VQ (Kumar, 2024). Residual VQ iteratively encodes the quantization error via multiple codebooks. ResApexQuant encodes the structural delta of the attention residual connection — a deterministic quantity with fixed geometric properties. No iterative codebook, no multi-stage encoding, no additional memory. The method is patent pending (INPI France, 2026).
Experiments and results
We evaluated ResApexQuant on the same benchmark setup as TurboQuant: real unit vectors on the sphere S^(d-1), dimensions d in {64, 128, 256}, bit-widths b in {1, 2, 3, 4}. We measure dot-product distortion (MSE) and 1@k recall ratio — the standard metrics for KV cache compression and vector search evaluation, as used in TurboQuant. All experiments are fully reproducible from the open-source code.
Distortion: MSE vs bit-width
TurboQuant achieves a distortion bound of sqrt(3*pi)/2 * 4^(-b) ~= 1.535 * 4^(-b), sitting 2.7x above the Shannon lower bound. ResApexQuant consistently sits far below TurboQuant at all bit-widths, with the largest gain at 1-bit where it achieves MSE = 0.03014 vs 0.36239 for TurboQuant — a factor of ×12.0.
| b | Method | MSE ↓ | Gain vs TurboQuant | Shannon lb |
|---|---|---|---|---|
| 1 | TurboQuant SOTA | 0.36239 | — | 0.25000 |
| 1 | ApexQuant | 0.17977 | +50.4% | |
| 1 | ResApexQuant OURS | 0.03014 | +91.7% | |
| 2 | TurboQuant SOTA | 0.11678 | — | 0.06250 |
| 2 | ApexQuant | 0.06179 | +47.1% | |
| 2 | ResApexQuant OURS | 0.03009 | +74.2% | |
| 3 | TurboQuant SOTA | 0.03423 | — | 0.01562 |
| 3 | ApexQuant | 0.01659 | +51.5% | |
| 3 | ResApexQuant OURS | 0.01016 | +70.3% | |
| 4 | TurboQuant SOTA | 0.00940 | — | 0.00391 |
| 4 | ApexQuant | 0.00610 | +35.1% | |
| 4 | ResApexQuant OURS | 0.00275 | +70.8% |
Vector search: 1@k recall ratio
Following TurboQuant's evaluation methodology on the GloVe dataset (d=200), we measure the 1@k recall ratio — how often the true nearest neighbor appears in the top-k compressed results. TurboQuant demonstrated superior recall over PQ and RabbiQ. ResApexQuant extends this advantage to a different magnitude.
Robustness across dimensions
KV head dimensions vary by architecture: d=64 for GPT-2, d=128 for LLaMA, d=256+ for larger models. We evaluate all methods across this full range. The delta ratio ||delta||/||h|| ≈ 0.41 is dimension-stable, making ResApexQuant architecture-agnostic.
Memory efficiency: same compression, better recall
ResApexQuant achieves its gains at identical memory cost. By using b-1 bits for delta_hat instead of b bits for h_hat, total compressed size is the same. The gain comes from operating on a geometrically simpler signal.
At 2-bit (8× vs FP16), ResApexQuant achieves Recall@1 = 0.710 — the same quality TurboQuant reaches only at 3-bit (5×). ResApexQuant delivers better results while using 37% less memory.
How ResApexQuant relates to TurboQuant
ResApexQuant builds on TurboQuant's theoretical contributions. TurboQuant established that Haar rotation + Lloyd-Max + QJL achieves near-optimal distortion. ResApexQuant adds two orthogonal innovations:
Innovation 1 — Fisher Saturation masking (ApexQuant). Constraining the Lloyd-Max codebook to [-Sc, Sc] where Sc = 0.769893 is the Fisher Saturation constant reduces distortion by ~50% over TurboQuant's unconstrained codebook. This applies to any vector quantization problem, independently of the residual approach.
Innovation 2 — Residual delta quantization (ResApexQuant). Compressing delta = h - x instead of h exploits the structural property ||delta||/||h|| ≈ 0.41, yielding an additional (0.41)^2 ≈ 6× Panter-Dite gain — and ×12 in practice at 1-bit.
On the theoretical side: TurboQuant proved a distortion bound of sqrt(3*pi)/2 * 4^(-b). ApexQuant improves this to Sc * sqrt(3*pi)/2 * 4^(-b). ResApexQuant achieves approximately Sc * (||delta||/||h||)^2 * sqrt(3*pi)/2 * 4^(-(b-1)) ≈ 0.125 * sqrt(3*pi)/2 * 4^(-b) — nearly 12× below TurboQuant's bound at 1-bit. These improvements are not heuristic; they follow from applying the Panter-Dite theorem to the actual signal being compressed.
Try It — Install & Test in 60 Seconds
ResApexQuant ships as a compiled C++/CUDA extension (.so). The mathematical core stays protected in the binary — you get the performance gains without the underlying IP being exposed. The CPU build is free for research. The production vLLM build requires an NDA.
CPU build
No GPU required. Benchmark on your laptop in under 60 seconds. Compiled C++ core, Python API exposed.
pip install pybind11 numpy
pip install -e resapex-lib/
resapex-bench --d 256 --all-bits
vLLM plugin
Drop-in for production deployments. Patches KV cache write/read in-place. Full CUDA kernel, no source exposed.
pip install resapex-lib[vllm]
from resapex import patch_vllm
patch_vllm(model, bits=2)
# 8× less KV cache, same quality
What’s inside the compiled .so
| Component | Free tier | Enterprise |
|---|---|---|
Python API — benchmark(), ResApexKVCache | ✓ Full source | ✓ Full source |
| Haar rotation + Lloyd-Max pipeline | Interface only | 🔒 Compiled |
| Fisher Saturation masking (Sc = 0.769893) | 🔒 Compiled | 🔒 Compiled |
| Residual delta calibration algorithm | 🔒 Compiled | 🔒 Compiled |
| QJL Fisher-weighted residual encoder | 🔒 Compiled | 🔒 Compiled |
| CUDA kernel — FP16/BF16 fast path (H100/A100) | — | 🔒 Compiled |
| Auto-calibration on real LLM KV vectors | — | 🔒 Enterprise only |
vLLM engine patching (patch_vllm()) | — | 🔒 Enterprise only |
These values are measured on 2,000 real unit vectors on Sd−1. The same benchmark protocol as TurboQuant (ICLR 2026).
Download resapex-lib (CPU)
Free for research and evaluation. Compiled C++ extension + Python API + CLI benchmark. Includes full test suite.
pip install pybind11 numpy && pip install -e .resapex-bench --d 256 --all-bitsBSL-1.1 — free for research · Commercial use requires a license
Request vLLM build + NDA
The production build includes the CUDA kernel, full vLLM engine patching, and auto-calibration on real KV vectors. Access requires an NDA. We reply within 24 h.
3 design partner slots open · Patent pending INPI France 2026
Request received!
We’ll send you the NDA within 24 hours.
In the meantime, download the CPU build to validate the benchmark numbers on your hardware.
Looking ahead
ResApexQuant demonstrates that substantial untapped structure exists in the residual connections of Transformer models. Every Transformer — GPT, LLaMA, Mistral, Gemini — uses residual connections. The delta delta = h - x is always smaller than h. This is a universal property.
More immediately, a production-grade implementation in vLLM would allow deploying LLaMA-3-70B at 1-bit KV cache with meaningful recall quality — something TurboQuant cannot achieve at this bit-width. On an A100 80GB with context length 128k, this would reduce KV cache memory from ~80GB to ~5GB, enabling true long-context inference at large scale.
ResApexQuant is data-oblivious, training-free, and online. It requires no fine-tuning, no architecture changes. Deployment requires replacing the KV cache write with a two-line delta computation followed by ApexQuant compression. The live benchmark is available on HuggingFace Spaces.
Labels
References
[1] Zandieh et al. TurboQuant. arXiv:2504.19874, ICLR 2026.
[2] Zandieh et al. QJL. arXiv:2406.03482, AAAI 2025.
[3] Hooper et al. KVQuant. NeurIPS 2024, arXiv:2401.18079.
[4] Kumar. Residual VQ for KV cache. arXiv:2410.15704, 2024.
[5] Panter & Dite. Quantization distortion. Proc. IRE, 1951.
[6] Lloyd. Least squares quantization. IEEE Trans. Inf. Theory, 1982.