Vector databases quantize embeddings to save memory, and the papers that introduce those quantizers, from product quantization [1] onward, report recall@k against an exact search. That is the right metric for a search engine, and it is how the field benchmarks itself [4]. It is an indirect metric for a retrieval-augmented generation (RAG) system [3], where the retrieved passages are consumed by a language model that may or may not notice a slightly worse ranking. ScaNN made a version of this argument one level down, measuring quantization error by its effect on ranking rather than on reconstruction [2]. hvec moves it one level up, to the answer.

Animated diagram of homomorphic compression in a RAG and agent-memory loop
The loop hvec measures. Memory and documents become f32 embeddings, a codec shrinks them, the query is scored directly on the compressed codes with no decode step, and the top passages enter the agent's context. The agent's new memory goes back through the same path. One-minute loop: the sequence plays in about thirteen seconds, then holds on the finished picture for forty-five before restarting; honours reduced-motion settings.

hvec exists to measure the direct thing: run the same questions through the same pipeline with different codecs and different chat models, and compare the answers, the latency, and the token cost. The benchmark is a matrix of embedding model × codec × chat model × dataset. Compression behaviour depends on the embedding model. Tolerance to degraded retrieval depends on the chat model. Both axes are recorded on every run.

What milestone 1 ships

  • A Cargo workspace with a Codec trait whose score method works on the compressed form. Only the f32 baseline exists so far; it is the thing every other codec will be measured against.
  • Connectors for Anthropic, any OpenAI-compatible server (OpenAI, Ollama, vLLM), and local ONNX embeddings (BGE small by default [6]), all behind named profiles in a TOML config.
  • A SQLite store that scores through the codec by brute force. Deliberately no index such as HNSW [5]: the point is to measure the codec, not the ANN structure.
  • An interactive shell. Type hvec, point it at a folder with /new docs ./docs, ask questions, switch models with /model, and every turn lands in a run log with timings, tokens, retrieved chunk ids and the prompt version.

What comes next

Int8 and binary codecs selectable per collection, then product quantization, then a bench command that runs a question set across the matrix and a report that summarises the log. The first published experiment will be int8 and binary against f32 on a few hundred QA pairs with a small local embedding model and two chat models. Generation-side scoring will start with exact match and add faithfulness in the RAGAS sense [7], graded by a cheaper model with the usual caveats about LLM judges [8].

Jargon used here is defined in the glossary.

If that sounds useful, the repository is at github.com/SwineCoder101/hvec and contributions are welcome.

References

  1. Hervé Jégou, Matthijs Douze, Cordelia Schmid. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(1), 2011. library
  2. Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar. Accelerating Large-Scale Inference with Anisotropic Vector Quantization. International Conference on Machine Learning (ICML), 2020. library
  3. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (NeurIPS), 2020. library
  4. Martin Aumüller, Erik Bernhardsson, Alexander Faithfull. ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms. Information Systems 87, 2020. library
  5. Yury A. Malkov, Dmitry A. Yashunin. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(4), 2020. library
  6. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff. C-Pack: Packed Resources For General Chinese Embeddings. arXiv:2309.07597, 2023. library
  7. Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert. Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217, 2023. library
  8. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems (NeurIPS), 2023. library

Terms used above are defined in the glossary.