The scheme

CKKS [1] is the homomorphic encryption scheme built for approximate arithmetic on real numbers, which is what a dot product of embeddings is. A ciphertext holds a vector of up to N/2 complex “slots” (so N/2 reals), where N is the ring dimension, and supports slot-wise addition and multiplication plus rotations of the slot vector. A dot product of two encrypted 384-dim vectors is one slot-wise multiply followed by log2(384) rotate-and-add steps to sum the slots. The result is an encrypted scalar; whoever holds the secret key decrypts it.

That is a homomorphic codec in exactly hvec’s sense: encode is encryption, score is the encrypted dot product, and decode is decryption. The shape fits the trait. The numbers do not.

Sizes

A CKKS ciphertext is two polynomials of degree N with coefficients modulo a product of primes q. For a dot product followed by one rescale, with the precision an embedding needs, a common parameter set is N = 8192 and log2(q) ≈ 200 bits, giving 2 × 8192 × 200 / 8 ≈ 410 KB per ciphertext [2]. One ciphertext packs N/2 = 4096 reals, so it can hold about 10 embeddings of 384 dimensions, or one with 3,712 dimensions of padding.

The first thing the encryption track does is invert hvec’s axis: every other codec makes the store smaller, this one makes it 27 to 270 times bigger. “Compression” is the wrong word, and the write-up should say so.

Time

Published library benchmarks put a CKKS ciphertext multiply with relinearisation and a rotation each at roughly a millisecond at N = 8192 on one core, with the exact figures depending on the library and the depth budget. A packed dot product against 10 stored vectors is one multiply and nine rotations: on the order of 10 ms. A brute-force scan of the 8,036-chunk SciFact collection is then about 800 packed operations, or several seconds per query. hvec’s f32 baseline scans the same collection in 4 ms, int8 in 2 ms.

Three orders of magnitude slower and 27× bigger, for exact scores. There is a fourth cost: the query must also be encrypted under the same key, and top-k selection cannot be done on encrypted scores without further machinery (comparison under CKKS is itself an approximate, multi-level circuit), so the server returns all encrypted scores and the client decrypts and ranks them. The bandwidth per query is then 8,036 encrypted scalars, which with packing is 800 ciphertexts, about 330 MB. That is the figure that ends the discussion for anything interactive.

What would make it interesting

The cost model is not hopeless, only mismatched with brute-force scan. Three changes move it:

• Ask a different question. The agent-memory loop writes often and reads rarely from a small store (hundreds of entries, not thousands). At 100 memories the scan is 10 packed operations and 10 ciphertexts back: tens of milliseconds and 4 MB. Encrypted memory for an agent whose host must not read its memories is a plausible setting.
• Quantize first, then encrypt. CKKS over sign bits or int8 codes needs far less precision, which means smaller q, smaller ciphertexts and shallower circuits. A SimHash-style [4] encrypted Hamming distance is a well-studied primitive. The codec composition hvec already has (mrl<k>+<codec>) is the right shape: enc+binary.
• Index, don’t scan. Clustering (as in IVF or PQ [3]) on plaintext centroids with encrypted residuals confines the encrypted work to one cluster. That leaks which cluster a query lands in, which is a privacy decision, not an engineering one.

Decision

No CKKS codec in hvec for now. The measurement harness is built around the claim that a codec’s cost is bytes and the question is whether the answers change; an encrypted codec’s cost is time and bandwidth and its answers do not change by construction, so almost nothing in bench recall or bench run would measure anything about it. If the track resumes, it starts with the agent-memory benchmark at a hundred entries and a quantize-then-encrypt codec, where the numbers above say the costs land within reach, and with a Codec extension that lets score return an opaque value a client decrypts, because the current trait’s f32 result is the one assumption an encrypted codec cannot meet.

References

1. Jung Hee Cheon, Andrey Kim, Miran Kim, Yongsoo Song. Homomorphic Encryption for Arithmetic of Approximate Numbers. ASIACRYPT, 2017. library
2. Craig Gentry. Fully Homomorphic Encryption Using Ideal Lattices. ACM Symposium on Theory of Computing (STOC), 2009. library
3. Hervé Jégou, Matthijs Douze, Cordelia Schmid. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(1), 2011. library
4. Moses S. Charikar. Similarity Estimation Techniques from Rounding Algorithms. ACM Symposium on Theory of Computing (STOC), 2002. library

Terms used above are defined in the glossary.

The objection is right about the premise and it sharpens the project rather than undermining it.

Layer one: the embedding compresses an operation, not bytes

Take a 200-word chunk. As UTF-8 it is roughly 1 KB. Its embedding from a 1536-dimension model is 1536 × 4 bytes, about 6 KB. In storage terms the embedding model has made the data six times larger. Nobody would call that compression.

What the embedding compresses is the cost of a question: are these two passages about the same thing? Answering that from text means reading and understanding both. Answering it from vectors is one dot product. The embedding model keeps exactly the information needed for that one operation and discards the rest, including, mostly, the ability to read the passage back. That is precisely the definition of homomorphic compression used throughout this project: preserve one operation on the compressed form, give up everything else.

“Mostly” is doing some work there. Inversion attacks have shown that short texts can be reconstructed from their embeddings almost word for word [1]. An embedding is a literal compressed representation of the passage, not an opaque fingerprint, and it should be handled with the same care as the text.

Layer two: quantization compresses the same operation and returns the bytes

hvec’s codecs take the vector the embedding model produced and preserve the same operation, inner product or cosine, while shrinking the representation 4× to 32×. Product quantization [2] is the canonical example: the stored vector becomes a handful of byte codes and the distance is computed by table lookups on those codes, never on a reconstructed vector.

So the stack is two homomorphic codecs in series, both preserving similarity. The interesting part is that their error models are different in kind.

• The embedding’s error is semantic. It is whatever the model thinks “similar” means, and it cannot be measured against the text in any clean way. There is no ground truth vector.
• The quantizer’s error is geometric. The f32 vector exists, so the quantizer’s score can be compared with the exact score for every pair. Recall against f32 and mean score error are well defined.

This is why the f32 baseline sits at the centre of hvec. It is the boundary between the two layers. Everything below it is measured exactly; everything above it is measured only through the answer.

The layers interact, which is why the embedding model is a benchmark axis

Quantization can only damage information the embedding kept. If a model spreads meaning evenly across its dimensions, a uniform int8 codec treats it fairly. If a model packs most of its meaning into a few leading dimensions, as Matryoshka-trained models deliberately do [3], the same codec spends precision on dimensions that carry little and the loss lands unevenly. Two models with the same dimension can react very differently to the same codec, and the right codec for one may be the wrong one for the other. ScaNN made a related point about measuring quantization by its effect on ranking rather than on reconstruction [4]; the same logic says you must measure it per embedding model.

That is the reason the benchmark matrix is embedding model × codec × chat model × dataset, and not codec alone.

The agent never reads a vector

Here is the part the objection gets slightly wrong, and it matters for what hvec claims to measure. In a retrieval-augmented pipeline [5] the vector is the key: it is what you search by. The chunk text is the payload: it is what comes back. The retrieval step scores the query against stored keys and returns the payloads for the best matches. The agent reads those payloads, as plain text, in its context window.

So no, the agent does not read from codecs instead of embeddings. It reads neither. Switching from f32 vectors to int8 codes changes how the keys are stored and how the scores are computed. It does not change the format of anything the model sees. What it can change is which passages come back, and that is the only route by which compression can alter the answer.

This is a useful boundary to draw. hvec compresses the key. Compressing the payload, by summarising or pruning passages before the model reads them [6], is a third layer that changes what the agent sees rather than what it finds. It is a legitimate axis with its own trade-offs, and it is out of scope for hvec today. Keeping the two apart makes the question hvec answers precise: does compressing the retrieval key change what gets retrieved enough to change the answer, and does that depend on which model is answering?

An experiment that falls out of this

If both layers compress the same operation, you can collapse them. Pick an embedding model that emits int8 or binary vectors natively and compare it against post-hoc quantization of an f32 model to the same byte budget, on the same questions, with the same chat model. If native low-precision wins, the right codec is “none, change layer one”. If post-hoc wins, quantization earns its place as a separate layer. Either result is worth publishing, and it will be one of the first experiments on this site.

Terms used here are defined in the glossary, including new entries for embedding as compression and key versus payload.

References

1. John X. Morris, Volodymyr Kuleshov, Vitaly Shmatikov, Alexander M. Rush. Text Embeddings Reveal (Almost) As Much As Text. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. library
2. Hervé Jégou, Matthijs Douze, Cordelia Schmid. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(1), 2011. library
3. Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ramanujan, William Howard-Snyder, Kaifeng Chen, Sham Kakade, Prateek Jain, Ali Farhadi. Matryoshka Representation Learning. Advances in Neural Information Processing Systems (NeurIPS), 2022. library
4. Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar. Accelerating Large-Scale Inference with Anisotropic Vector Quantization. International Conference on Machine Learning (ICML), 2020. library
5. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (NeurIPS), 2020. library
6. Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, Lili Qiu. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. library

Terms used above are defined in the glossary.

hvec exists to measure the direct thing: run the same questions through the same pipeline with different codecs and different chat models, and compare the answers, the latency, and the token cost. The benchmark is a matrix of embedding model × codec × chat model × dataset. Compression behaviour depends on the embedding model. Tolerance to degraded retrieval depends on the chat model. Both axes are recorded on every run.

What milestone 1 ships

• A Cargo workspace with a Codec trait whose score method works on the compressed form. Only the f32 baseline exists so far; it is the thing every other codec will be measured against.
• Connectors for Anthropic, any OpenAI-compatible server (OpenAI, Ollama, vLLM), and local ONNX embeddings (BGE small by default [6]), all behind named profiles in a TOML config.
• A SQLite store that scores through the codec by brute force. Deliberately no index such as HNSW [5]: the point is to measure the codec, not the ANN structure.
• An interactive shell. Type hvec, point it at a folder with /new docs ./docs, ask questions, switch models with /model, and every turn lands in a run log with timings, tokens, retrieved chunk ids and the prompt version.

What comes next

Int8 and binary codecs selectable per collection, then product quantization, then a bench command that runs a question set across the matrix and a report that summarises the log. The first published experiment will be int8 and binary against f32 on a few hundred QA pairs with a small local embedding model and two chat models. Generation-side scoring will start with exact match and add faithfulness in the RAGAS sense [7], graded by a cheaper model with the usual caveats about LLM judges [8].

Jargon used here is defined in the glossary.

If that sounds useful, the repository is at github.com/SwineCoder101/hvec and contributions are welcome.

References

1. Hervé Jégou, Matthijs Douze, Cordelia Schmid. Product Quantization for Nearest Neighbor Search. IEEE Transactions on Pattern Analysis and Machine Intelligence 33(1), 2011. library
2. Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, Sanjiv Kumar. Accelerating Large-Scale Inference with Anisotropic Vector Quantization. International Conference on Machine Learning (ICML), 2020. library
3. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. Advances in Neural Information Processing Systems (NeurIPS), 2020. library
4. Martin Aumüller, Erik Bernhardsson, Alexander Faithfull. ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms. Information Systems 87, 2020. library
5. Yury A. Malkov, Dmitry A. Yashunin. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Transactions on Pattern Analysis and Machine Intelligence 42(4), 2020. library
6. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muennighoff. C-Pack: Packed Resources For General Chinese Embeddings. arXiv:2309.07597, 2023. library
7. Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert. Ragas: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217, 2023. library
8. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, Ion Stoica. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems (NeurIPS), 2023. library

Terms used above are defined in the glossary.