The Vector Database You Were About to Buy Is a For Loop

Every post in this series so far has been about generation: the model that writes the summary, translates the sentence, answers the question. That is the half everyone demos. The half that actually decides whether the answer is any good is retrieval, which is the unglamorous business of finding the right three paragraphs to hand the model before it opens its mouth.

Retrieval is also where the architecture diagrams get expensive. The standard recipe says you call an embedding API for every document you own, push the resulting vectors into a hosted vector database, and then call the embedding API again for every query, forever. Three line items, two of them metered, one of them a subscription. For a corpus the size most small operations actually have, all three are optional.

The embeddings can be computed in the browser. The search is a loop. Your documents never leave the machine.

What retrieval actually is, stripped of the vocabulary

An embedding model turns a piece of text into a list of numbers, positioned so that text with similar meaning lands nearby. "How do I get my money back" and "refunds are issued to the original payment method" share almost no words, but their vectors sit close together, which is exactly the trick keyword search cannot do.

Semantic search is then three steps, none of them exotic:

1. Cut your documents into chunks.
2. Turn every chunk into a vector once, and keep the vectors.
3. Turn the query into a vector, and find the chunks whose vectors point the most similar direction.

Step three is the one the vector database is sold to solve. If your vectors are normalized to unit length, "similarity" is the dot product: multiply the two lists element by element and add up the results. For a 384-dimensional vector that is 384 multiplications. Over two thousand chunks it is 768,000 multiplications, which a phone does without noticing.

That is the whole comparison engine. A vector database earns its money when you have millions of vectors and need an approximate index to avoid scanning them all. At a few thousand, scanning them all is faster than the network round trip you were going to make to ask something else to do it.

The models got small enough to live in a tab

The reason this is a 2026 conversation and not a 2023 one is that the embedding models shrank into browser range while staying good.

The flagship case is Google's EmbeddingGemma, released in September 2025: a 308 million parameter embedding model built on Gemma 3, trained on 100+ languages, that Google positions as "the highest ranking open multilingual text embedding model under 500M" on the Massive Text Embedding Benchmark, and describes as "small enough to run on less than 200MB of RAM with quantization" (Google Developers Blog). It has a 2K token context window and outputs 768-dimensional vectors, and thanks to Matryoshka representation learning you can truncate those vectors to 512, 256, or 128 dimensions and keep most of the quality, which cuts your storage and your comparison cost proportionally (model card).

That is the headline model, and it is worth knowing about. It is not, however, what I reach for first in a browser, and I want to be straight about why. The ONNX build lives at onnx-community/embeddinggemma-300m-ONNX and it is not gated, but the 4-bit weights are still around 188 MB split across an external data file, it ships under the Gemma license rather than a plain open-source one, it needs specific task prefixes on every input ("task: search result | query: " for queries, "title: none | text: " for documents), and the model card's own JavaScript example drives it through AutoModel rather than the feature-extraction pipeline. That is a fine set of tradeoffs for a desktop app. It is a lot of first-load weight for a web page.

The pragmatic browser default is still the small BERT-family retrieval models, and the sizes are the argument. These are the real quantized ONNX file sizes, read off the Hugging Face Hub rather than estimated:

Model	Quantized ONNX	Dimensions
all-MiniLM-L6-v2	22,972,370 bytes (~22 MB)	384
bge-small-en-v1.5	34,014,426 bytes (~32 MB)	384
snowflake-arctic-embed-s	34,015,111 bytes (~32 MB)	384

Twenty-two megabytes. That is a mid-sized hero image. It downloads once, the browser caches it, and every embedding after that is free forever.

The library, and the two defaults that will bite you

Transformers.js is what runs these in the browser. It hit version 4 in February 2026 with a WebGPU runtime rewritten in C++ alongside the ONNX Runtime team (release post), and the current published package is 4.2.0.

The API is four lines, and two of its defaults are quiet traps.

import { pipeline } from '@huggingface/transformers';

const embed = await pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2', {
  device: 'webgpu',  // omit or use 'wasm' for the CPU path
  dtype: 'q8',       // <- trap #1. See below.
});

// One vector per input. Batch your chunks; do not loop one at a time.
const vectors = await embed(chunks, { pooling: 'mean', normalize: true });
//                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ trap #2

Trap one: dtype decides which file you download. Transformers.js maps q8 to the model_quantized.onnx file and fp32 to the plain model.onnx. If you do not pass dtype, the library picks a default per device: q8 on WASM, and fp32 on everything else, WebGPU included (dtype source). So the innocent-looking device: 'webgpu' with no dtype quietly fetches the 86 MB fp32 MiniLM instead of the 22 MB quantized one. Name the dtype.

Trap two: pooling defaults to 'none'. The feature-extraction pipeline's documented default is no pooling and no normalization, which hands you the raw per-token hidden states, a tensor shaped like [1, tokens, 384], not a sentence vector (pipeline source). You want pooling: 'mean' and normalize: true. And the right pooling depends on the model: MiniLM is a mean-pooling model, while bge and arctic-embed are CLS-pooling models, so pass pooling: 'cls' for those. The pipeline accepts 'none', 'mean', 'cls', 'first_token', 'eos', and 'last_token'.

With normalize: true, cosine similarity collapses into a dot product, and the search half of your RAG stack is this:

function dot(a, b) {
  let s = 0;
  for (let i = 0; i < a.length; i++) s += a[i] * b[i];
  return s;
}

const q = await embed([query], { pooling: 'mean', normalize: true });
const qv = q.tolist()[0];

const ranked = vectors
  .map((v, i) => ({ i, score: dot(qv, v) }))
  .sort((a, b) => b.score - a.score)
  .slice(0, 5);

That is it. That is the part you were quoted a monthly subscription for.

About sqlite-vec, honestly

The obvious question is whether you should reach for a real embedded vector store instead of an array. The usual answer is sqlite-vec, which is a genuinely good project: pure C, no dependencies, stores vectors in vec0 virtual tables, does KNN search in plain SQL, and runs anywhere SQLite runs.

The browser story is rougher than the pitch suggests, and you should know that before you plan around it. It is pre-1.0 and says so: the current line is v0.1.9 stable with v0.1.10 in alpha, and the documentation still carries a work-in-progress banner. More to the point for us, you cannot load it into a browser at all in the normal way. The project's own WASM page states plainly that "it's not possibly to dynamically load a SQLite extension into a WASM build of SQLite", so sqlite-vec "must be statically compiled into custom WASM builds", and the published sqlite-vec-wasm-demo package is explicitly labelled "a demonstration and may change at any time" that does not follow the project's semantic versioning (sqlite-vec WASM docs).

Translated: to use sqlite-vec in a browser today you compile your own SQLite WASM build, or you depend on a demo package that tells you not to depend on it. Both are real options for someone who wants persistence and SQL over their vectors. Neither is a thing you casually add on a Tuesday.

So here is the honest hierarchy. Under a few thousand chunks, use an array and a loop, and put the vectors in IndexedDB if you want them to survive a reload. Past that, look at a real index. Do not skip the array stage because a diagram told you to.

Reranking, if you want the extra accuracy

One upgrade is worth naming because it also runs locally. Embedding search compares two vectors that were computed without ever seeing each other, which is fast but blunt. A cross-encoder reads the query and the candidate chunk together and scores the pair directly, which is much better and much slower. The standard pattern is to retrieve the top 20 or 30 with embeddings, then rerank just those with the cross-encoder.

A real ONNX cross-encoder does exist for the browser: Xenova/ms-marco-MiniLM-L-6-v2 is the ONNX conversion of the standard cross-encoder/ms-marco-MiniLM-L-6-v2 reranker, its quantized weights are 23,143,499 bytes (about 22 MB), and it runs through Transformers.js as a sequence-classification model. So a full retrieve-then-rerank pipeline is roughly 45 MB of one-time download and still zero per-query cost. Whether the second 22 MB earns its place is a judgment call about your corpus, not a foregone conclusion.

Where this breaks

The same shape of caveats as the rest of the series, and they are real.

• The first load is a real download. Twenty-two megabytes is small for a model and large for a web page. Gate it behind an explicit click, show progress, and never fetch it just because someone opened the page. The second visit is free because the browser cached it; the first visit is not.
• Indexing is not instant. Embedding a thousand chunks is a thousand forward passes. It runs, and it is quick enough to sit through on a decent machine, but it is not free of time even though it is free of money. If the corpus is yours rather than the visitor's, embed it at build time and ship the vectors as JSON.
• WebGPU is wide, not universal. It reached Baseline in January 2026 and ships by default in Chrome, Edge, Firefox, and Safari 26 (web.dev); caniuse currently puts full support around 82% of tracked users, or about 84% counting partial (caniuse). Transformers.js falls back to WebAssembly on the CPU on its own, so the feature still works without a GPU. It is just slower, which for embedding a short query is fine and for indexing a large corpus is not.
• There is no built-in embedding API to fall back on. Chrome's stable built-in AI set is Summarizer, Translator, and Language Detector, with the Prompt API still on an origin trial for the web (Chrome built-in AI APIs). None of them is an embedder. Unlike the summarize-and-translate story, there is no shortcut here where the browser already has the model; you download one or you call a server.
• Memory is the ceiling, as always. The weights, the runtime, and every vector you keep all live in the tab. Vectors are cheap (a 384-dimension float32 vector is about 1.5 KB, so ten thousand chunks is roughly 15 MB), but the model plus the runtime is the floor you cannot go under.

Try it before you build it

I built the sandbox version of this so you can watch it happen rather than take my word for it.

Try the Client-Side Semantic Search Sandbox: paste your own text or load the sample corpus, and it chunks it, embeds every chunk on your own hardware, and ranks the chunks against your query by cosine similarity, showing the score for each. It detects WebGPU and falls back to the CPU with a message rather than breaking. The model download is behind a button and nothing happens until you press it. Nothing you paste is uploaded, because there is no server call in the page at all.

The thing to actually try: load the sample corpus and ask "my card is closed, how do I get my money back". The top hit is the refund paragraph, which never uses the words "closed" or "money". That gap between the words you typed and the words in the answer is the entire reason embeddings exist, and it just ran on your laptop for nothing.

The bottom line

Retrieval-augmented generation got sold as an architecture. For most of the corpora that small operations actually own, it is a 22 MB download and a for loop. The embedding model runs on the visitor's chip, the vectors sit in an array, the search is a dot product, and the documents, which are usually the sensitive part, never leave the machine at all. You pay once in download bytes and nothing per query after that, which is the same floor this whole series keeps landing on.

Buy the vector database when you have the vectors to justify it. Until then, you are paying a subscription to run a loop you could have written.

If your reflex when you need a capability is to price out a monthly tool for it, and you would rather that reflex were "what does this cost me if I just build it", that swap is the argument of my book The $20 Dollar Agency (search the title on Amazon Kindle). The short version lives on this blog, in the posts below.

Related reading

• You Can Now Run Real AI in the Browser for Free: the anchor post for this series, and the vision-and-audio side of the same shift.
• When the Built-in LLM Won't Reach: the bring-your-own-model path that Transformers.js belongs to, and its tradeoffs.
• Delete Your Translation Bill: the stable Chrome task APIs, and the reason there is no built-in embedder to lean on here.
• Your Browser Has a Free LLM Now: the generation half that the retrieval in this post is meant to feed.
• WebGPU Everywhere: why the GPU in the visitor's machine became a compute target you can plan around.

Fact-check notes and sources

• EmbeddingGemma specs: 308 million parameters, released September 4 2025, built on Gemma 3, trained on 100+ languages, a 2K token context window, output dimensions customizable "from 768 to 128" via Matryoshka representation learning, "small enough to run on less than 200MB of RAM with quantization", and described as "the highest ranking open multilingual text embedding model under 500M on the Massive Text Embedding Benchmark" (Google Developers Blog, model card). The model card lists MRL truncation options of 512, 256, and 128 and a Gemma license.
• EmbeddingGemma in the browser: the ONNX build at onnx-community/embeddinggemma-300m-ONNX is not gated, offers fp32, q8, and q4 (the card notes activations do not support fp16), requires the "task: search result | query: " and "title: none | text: " prefixes, and its q4 weights total roughly 188 MB across an external data file. Sizes read from the Hugging Face Hub file listing on 2026-07-15.
• Model download sizes: quantized ONNX weights measured from Hugging Face Hub on 2026-07-15: all-MiniLM-L6-v2 22,972,370 bytes (fp32: 90,387,606), bge-small-en-v1.5 34,014,426 bytes, snowflake-arctic-embed-s 34,015,111 bytes, ms-marco-MiniLM-L-6-v2 23,143,499 bytes. All three embedding models report hidden_size 384 and max_position_embeddings 512 in their configs.
• Transformers.js v4: released February 9 2026 with a WebGPU runtime rewritten in C++ with the ONNX Runtime team; current published package version 4.2.0 (release post, npm).
• Pipeline defaults: feature-extraction defaults to pooling: 'none' and normalize: false, and accepts 'none', 'mean', 'cls', 'first_token', 'eos', 'last_token' (pipeline source). The dtype-to-filename map (q8 to _quantized, fp32 to no suffix) and the per-device dtype defaults (q8 on WASM, fp32 otherwise) are in utils/dtypes.js.
• sqlite-vec status: pre-1.0, v0.1.9 stable with v0.1.10 in alpha, documentation flagged work-in-progress; stores vectors in vec0 virtual tables and does KNN in SQL (sqlite-vec, releases). Browser use requires static compilation into a custom SQLite WASM build because "it's not possibly to dynamically load a SQLite extension into a WASM build of SQLite"; the sqlite-vec-wasm-demo package is "a demonstration and may change at any time" and does not follow sqlite-vec's semantic versioning (sqlite-vec WASM docs).
• In-browser cross-encoder reranking: Xenova/ms-marco-MiniLM-L-6-v2 is an ONNX conversion of cross-encoder/ms-marco-MiniLM-L-6-v2 for Transformers.js, used as a sequence-classification model over query and document pairs.
• WebGPU support: Baseline as of January 2026, shipping by default in Chrome and Edge 113+, Firefox 141+, and Safari 26 (web.dev); caniuse reports 82.17% full support plus 1.46% partial (caniuse).
• No built-in browser embedding API: Chrome's built-in AI set covers Summarizer, Translator, and Language Detector as stable, with the Prompt API on an origin trial for the web, and does not include an embedding API (Chrome built-in AI APIs).
• Not claimed here: no in-browser embedding throughput or latency benchmark is given, because I did not measure one across a representative device range. The sandbox reports the real time your own machine takes, which is the only number worth trusting.

---

This post is informational, not engineering advice. Mentions of Google, Hugging Face, Snowflake, Microsoft, SQLite, and other third parties are nominative fair use. No affiliation is implied. Library versions, model availability, file sizes, and browser support change quickly; verify against the current project documentation before you build.