Full write-up, raw results and a one-command reproduction are in the repository: experiments/2026-10-scifact-int8-binary.

Hypothesis. Scalar int8 quantization will be close to lossless for retrieval on real scientific-abstract embeddings, and binary (sign) quantization will lose a large fraction of neighbours, by an amount that depends on the embedding model.

Result. Confirmed. 8,036 chunks from BEIR SciFact [5], 300 real test queries, cosine, brute force, no rescoring:

embedding model codec recall@10 top-1 mean score error bytes ratio
bge-small-en-v1.5 [4] int8 0.987 0.997 0.0005 392 3.9×
bge-small-en-v1.5 binary 0.611 0.700 0.307 48 32×
all-MiniLM-L6-v2 int8 0.995 1.000 0.0003 392 3.9×
all-MiniLM-L6-v2 binary 0.705 0.730 0.034 48 32×

Recall@100 for binary is 0.604 and 0.708: the lost neighbours are reordered far away, not just past the cutoff.

Reading. int8 is close to free on both models. Binary behaves very differently per model in score error (BGE vectors are not centred, so many sign bits agree across the whole corpus and discriminate nothing) but only moderately differently in recall. Sign quantization loses rank information even when scores track well, which is the gap that learned rotations [3] and error-bounded binary schemes [6] exist to close. The asymmetric sign score is the SimHash estimator [2]; product quantization [1] is the next codec in line.

Caveats. One corpus, two small models, no relevance labels, no chat model. Nothing here says whether 0.61 recall changes an answer. That is the next experiment.

Follow-up. The centring step suggested above is measured in SciFact: mean-centred binary against plain binary: it fixes the scores and recovers only four or five points of recall.

References

  1. Hervé Jégou, Matthijs Douze, Cordelia Schmid. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(1), 2011. library
  2. Moses S. Charikar. Similarity Estimation Techniques from Rounding Algorithms. ACM Symposium on Theory of Computing (STOC), 2002. library
  3. Yunchao Gong, Svetlana Lazebnik, Albert Gordo, Florent Perronnin. Iterative Quantization: A Procrustean Approach to Learning Binary Codes for Large-Scale Image Retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence 35(12), 2013. library
  4. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff. C-Pack: Packed Resources For General Chinese Embeddings. arXiv:2309.07597, 2023. library
  5. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, Iryna Gurevych. BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models. NeurIPS Datasets and Benchmarks, 2021. library
  6. Jianyang Gao, Cheng Long. RaBitQ: Quantizing High-Dimensional Vectors with a Theoretical Error Bound for Approximate Nearest Neighbor Search. Proceedings of the ACM on Management of Data (SIGMOD), 2024. library

Terms used above are defined in the glossary.