# Fitting RAG in Your Pocket: Local Retrieval in React Native
> Build a complete Retrieval-Augmented Generation pipeline that never leaves the phone: local embeddings, native vector search with nitro-sqlite + sqlite-vec, and local generation using React Native.
Canonical article: https://margelo.com/blog/fitting-RAG-in-your-pocket
LLM text: https://margelo.com/blog/fitting-RAG-in-your-pocket/llms.txt
Author: Ritesh Shukla
Category: Engineering
Tags: React Native, RAG, Vector Search, On-Device AI, SQLite
Published: 2026-08-10
Last updated: 2026-08-10
Reading time: 23 minutes
## Article
Open almost any app today and there's probably an AI chatbot that already knows you. Ask your food-delivery app *"what did I order last Friday?"* and it answers from the history of **your** orders, not the whole menu.
I know some of you might have wondered: "What sort of sorcery is that? How is it possible?" - but no, it's not magic, it's something called RAG.
> Retrieval-Augmented Generation (RAG) is the process of optimizing the output of a large language model, so it references an authoritative knowledge base outside of its training data sources before generating a response. _[Source](https://aws.amazon.com/what-is/retrieval-augmented-generation/)_
Let's build something similar using React Native, but for your own chats: export a **WhatsApp conversation**, feed it to an AI, so you can ask questions in plain language and get answers from the actual messages. The one rule: everything runs **on device**. No server, no API key, no per-token bill; on a $100 Android phone in airplane mode, it still works.
> If you want an example of cloud-based RAG, consult this other [blog post](https://margelo.com/blog/building-native-llm-chat-app-with-rag#answering-margelo-questions-with-rag) authored here at Margelo!
Ready? Let's go!
## RAG needs a brain ๐ค
We start from a basic chat screen: a message list, an input box, and a send handler that hands the conversation to a model and renders the reply. (We built exactly this in [a previous post](https://margelo.com/blog/building-native-llm-chat-app-with-rag), if you're interested in the UI side.) In a typical AI chat app that model lives on a server, and that's the easy version: one `fetch` and you're done. But it breaks our one rule, so the first move is swapping the server model for one that runs on the device.
Generation runs on a small on-device LLM: [Qwen2.5-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2.5-0.5B-Instruct-GGUF), a 4-bit GGUF loaded through [`@react-native-ai/llama`](https://github.com/callstackincubator/ai), which wraps llama.cpp. It's deliberately small, 469 MB at Q4, because it has to fit inside a cheap phone's memory, not a datacenter. Loading it is one call:
```ts
import { getModelPath, llama } from '@react-native-ai/llama'
const MODEL = 'Qwen/Qwen2.5-0.5B-Instruct-GGUF/qwen2.5-0.5b-instruct-q4_k_m.gguf'
const qwen = await llama.languageModel(getModelPath(MODEL)).prepare()
```
Now, how does the chat get in front of it? The obvious _albeit rookie_ way: read the whole WhatsApp export, give it to our LLM, and then ask the question.
```ts
const chat = await loadWhatsAppExport() // the entire _chat.txt, as one string
const { text } = await qwen.completion({
messages: [
{ role: 'system', content: 'Answer questions about this WhatsApp chat.' },
{ role: 'user', content: `${chat}\n\nQuestion: ${question}` },
],
})
```
The model reads the whole conversation as its context and answers from it. Simple. Right?
Testing our hypothesis
Let's load a real export, ask a plain question, and see what comes back. We'll use a WhatsApp export, a few months of everyday chat between two friends: work banter, cricket, a weekend trip that got planned, cancelled, and re-planned across dozens of messages. Then we ask something genuinely buried:
> Who ended up paying for the hotel?
Not a date lookup. The chat is full of hotel talk, two different hotels got booked and one got cancelled, and the messages that actually answer this say *"I put the room on my card"* and talk about *"the cottage"*, never the word "hotel". We'll come back to why that matters.
Oops. The model never even gets to answer. Why?
Every LLM can only read so much at once: its **context window**, measured in tokens. Qwen2.5-0.5B can natively stretch to **32,768** tokens, but a phone can't afford that window. It is paid for in RAM, the **KV cache** the model holds for every token in the window, and that cost grows as the window does. With the 469 MB model already resident in ~900 MB of usable memory, and headroom needed for machinery we'll add later, we cap the context at **4,096**. Our WhatsApp export is over **21,000** tokens, so it doesn't fit. Not "slow," not "truncated": *rejected outright*. llama.cpp refuses the request and reports `Context is full`, and the model never sees a single message.
> The context window (or "context length") of a large language model (LLM) is the amount of text, in tokens, that the model can consider or "remember" at any one time. _[Source](https://www.ibm.com/think/topics/context-window)_
So what are the options? **A bigger window?** The window *is* that KV cache; 32K of it won't fit next to the model in this phone's RAM, and a year of messages sails past 32K anyway. **A bigger model?** Worse: more weights leave even less RAM for the window, until the process just dies. **A cloud model?** The whole chat fits, but the messages leave the phone, it stops working offline, and you re-send all 21,000 tokens, paid per token, on every single question.
The real fix: send less, not more
Every option above tries to fit the *whole* chat somewhere, a bigger window, a bigger machine, someone else's server. But look at the question again: *"Who ended up paying for the hotel?"* The answer is one or two messages out of thousands. We never needed the whole chat in context. We needed those two lines.
So instead of dumping everything, we **retrieve** the few relevant messages first, then hand *only those* to the model. That's where **RAG** comes in: look the facts up before you answer, and answer from what you found.
The hard part is those few relevant messages. Given a question you've never seen before, how do you find the lines that answer it?
## Building the retrieval pipeline
Finding those lines is a pipeline of its own, so let's build it a piece at a time.
Retrieval by meaning: vector search
Let's take it one step at a time. First idea: split the chat into small **chunks**, a few messages each, and for every question pick only the chunks that could answer it. The plan is sound; the whole problem collapses into *which chunks?*
The obvious attempt is keyword matching: grep the chunks for the words in the question. It falls apart on the first real query. Someone asks about the "electrician" when the message says *"the guy coming to fix the lights,"* or types "hackathon result" when the chat says "secured 2nd place." Match on words and you miss every synonym, every nickname, every switch of language. You have to match on **meaning**.
We need something more solid, and that's where **embeddings** come in. The word is heavier than the idea: represent each chunk as a list of numbers, a **vector**. Picture it in 3D first: a vector is just three coordinates, a point with a direction, and we arrange things so chunks with similar meaning point close together, *"the guy fixing the lights"* near "electrician", unrelated text far apart.
A real embedding model does exactly this, just in hundreds of dimensions instead of three, and it has already learned which texts belong near each other, so we don't touch the math. Embed the question with the same model and search stops being string-matching and becomes geometry: find the chunk vectors nearest the question's vector.
Here's the whole pipeline, ingest on top and query on the bottom, meeting at the database:
Walking through each stage:
1. **Documents:** the raw data: here, the WhatsApp messages.
2. **Chunking:** each document split into chunks. Smaller retrieves more precisely, but overdo it and you lose surrounding context.
3. **Embedding:** each chunk turned into a vector (curious how? Callstack's [primer on local embeddings](https://www.callstack.com/blog/on-device-ai-introducing-apple-embeddings-in-react-native) is a great start).
4. **Vector database:** stores the vectors, runs the similarity searches.
5. **Query embedding:** the question through the same model.
6. **Similarity metric:** the closeness math (cosine similarity, dot product).
7. **Nearest neighbour search:** the closest chunk vectors to the query.
8. **Results:** the matching chunks, ready for the LLM (each chunk's text is stored next to its vector, so a hit carries the original text with it).
Turning SQLite into a vector database
The whole pipeline hinges on one component being fast on a phone: the vector database sitting in the middle of that diagram. At Margelo we have built a number of offline-first apps. That demands a fast local database, which is why we built [`react-native-nitro-sqlite`](https://github.com/margelo/react-native-nitro-sqlite), an SQLite library that talks to native SQLite over an optimized JSI pipeline (ie, [nitro](https://nitro.margelo.com)) with almost no overhead.
As local AI took off, these offline-first apps needed vectors. A separate vector store would mean a second database to keep in sync with the first, its own query language, and another engine baked into the app binary. So rather than bolt on a separate store, we compiled [`sqlite-vec`](https://github.com/asg017/sqlite-vec) **straight into nitro-sqlite**. The result is a vector database *inside* the same SQLite file your app already uses: no service, no runtime extension, KNN running natively over JSI. If you're curious how we wired `sqlite-vec` into the build, [this PR](https://github.com/margelo/react-native-nitro-sqlite/pull/285) has the full story.
Enabling the vector database
Install and rebuild the native app:
```sh
npm install react-native-nitro-sqlite react-native-nitro-sqlite-vec react-native-nitro-modules
# iOS: the NITRO_SQLITE_VEC flag tells the pod to compile sqlite-vec in
NITRO_SQLITE_VEC=1 npx pod-install
# Android: add `nitroSqliteVec=true` to android/gradle.properties
# rebuild: these are native modules, a JS reload is not enough
npm run ios # or: npm run android
```
Once the flag is set, `sqlite-vec` is **statically linked** into nitro-sqlite's SQLite: nothing to fetch, nothing to load at runtime. Confirm it's live from JS:
```ts
import { open } from 'react-native-nitro-sqlite'
const db = open({ name: 'ragchat' })
const { rows } = db.execute('SELECT vec_version() AS v')
console.log('sqlite-vec', rows?._array?.[0]?.v) // e.g. "v0.1.9"
```
Creating the table
sqlite-vec adds a `vec0` virtual table: one fixed-size vector column plus *auxiliary* columns (prefixed `+`) that ride along with each row and come back in your KNN results. So we store the chunk text and source next to the vector, no JOIN needed:
```ts
db.execute(`
CREATE VIRTUAL TABLE IF NOT EXISTS chunks USING vec0(
embedding float[384] distance_metric=cosine, -- must match the model's output size
+content text, -- the chunk we'll feed back to the LLM
+source text -- where it came from
);
`)
```
That `384` is the output dimension of the embedding model, and column width and model must agree. In a real app, name the table after the *embedder*, not the dimension (`chunks_minilm`, not `chunks_384`): two 384-dimension models produce vectors that are equally valid and mutually incomparable, so a width-keyed table will silently accept vectors from a model you swapped out last week. No error, just a wrong answer waiting downstream. (We'll keep the short `chunks` here so the snippets stay readable.)
Chunking the conversation
Chunk too aggressively and you tear a fact from its subject; don't chunk and you blow the context window. A chat log has a natural unit that a PDF doesn't: **a burst of messages on one day**. So we pack consecutive messages into small windows, and force a new window whenever the day changes, so two unrelated threads never share a chunk.
Each line keeps its sender, because "who said it" is half of what you'll want to ask:
```ts
const MAX_CHARS = 200 // a window: a few messages of back-and-forth
const HARD_MAX_CHARS = 600 // no single line may exceed this, see below
function chatToChunks(messages: Message[]): string[] {
const chunks: string[] = []
let buf: string[] = []
let day = ''
const flush = () => { if (buf.length) chunks.push(buf.join('\n')); buf = [] }
for (const m of messages) {
// Split the *message*, not the finished window: every part must keep its
// "Sender:" prefix, or the tail of a long message arrives with no speaker.
for (const part of splitLong(m.text, HARD_MAX_CHARS)) {
const line = `${m.sender}: ${part}`
if (buf.length && (m.date !== day || buf.join('\n').length + line.length > MAX_CHARS)) flush()
day = m.date
buf.push(line)
}
}
flush()
return chunks
}
```
That comment cost me an evening. My first version split the finished *window*, so a 3,900-character trip itinerary got cut in half and the second half lost its `Ayush:` prefix. Ask about anything living in that tail, "who has the printed tickets?", and the model, with no speaker in the context, confidently attributes it to the *other* person. The retrieval was perfect; the chunker had thrown the answer away.
> Chunk boundaries are not a formatting detail, they're part of your data model. Anything you strip at chunk time (speaker, timestamp, thread) is something the model will cheerfully hallucinate later. The embedding step coming up can lose a fact just as quietly, in a different way.
Embedding on the device
This is the heart of the pipeline: turning a chunk of text into a vector of 384 numbers, on the device.
You don't need a second ML runtime for this: llama.cpp runs an embedding model as happily as a chat model, so the same engine that generates our answers also produces our vectors, one dependency, one set of build flags. Load the model with `embedding: true` and call `.embedding()`:
```ts
import { getModelPath, llama } from '@react-native-ai/llama'
import type { LlamaContext } from 'llama.rn'
const EMBED_MODEL = 'mykor/paraphrase-multilingual-MiniLM-L12-v2.gguf/paraphrase-multilingual-MiniLM-L12-118M-v2-Q4_K_M.gguf'
const DIM = 384
// Cache the *promise*, not the context: a second call mid-load awaits the
// same load instead of orphaning a ~120 MB native context.
let ctxPromise: Promise | null = null
function embedder() {
ctxPromise ??= llama.textEmbeddingModel(getModelPath(EMBED_MODEL), {
normalize: 2, // L2, so cosine == dot product
contextParams: { embedding: true, n_ctx: 4096, n_gpu_layers: 0, n_threads: 4 },
}).prepare()
return ctxPromise
}
export async function embed(texts: string[]): Promise {
const c = await embedder()
const out: number[][] = []
for (const text of texts) {
const { embedding } = await c.embedding(text, { embd_normalize: 2 })
out.push(embedding) // already mean-pooled and normalized
}
return out
}
```
The model is [**paraphrase-multilingual-MiniLM-L12-v2**](https://huggingface.co/sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2): 384 dimensions, ~119 MB at Q4, trained on 50+ languages. I reached for a plain English `all-MiniLM-L6` first, since it's smaller and it's what everyone benchmarks. Then I pointed it at my own real export, which switches languages mid-sentence, and it was useless: messages outside English collapsed to nearly the same vector and every search came back noise. If there is any chance your users' chats aren't pure English, the multilingual model is worth every megabyte.
> The same `embed()` must run over your documents *and* the query. Different models, or even different normalization, put the vectors in different spaces and search returns nonsense. This is the number one RAG pitfall, and it's locked in before a single vector reaches storage.
Storing the vectors
Chunk, embed, batch insert. sqlite-vec takes the vector as a JSON array string, and `executeBatch` writes every chunk in one native round trip:
```ts
export async function addConversation(messages: Message[], source: string) {
const chunks = chatToChunks(messages)
const vectors = await embed(chunks)
db.executeBatch(
chunks.map((content, i) => ({
query: 'INSERT INTO chunks(embedding, content, source) VALUES (?, ?, ?)',
params: [JSON.stringify(vectors[i]), content, source],
})),
)
}
```
The conversation is now searchable on the device; call `addConversation` again to add another chat.
Answering a question
Now the query side. Say we ask: *"Who ended up paying for the hotel?"*
**Embed the query** with the *same* model, then **run the KNN search**. sqlite-vec's `MATCH ... AND k = ?` does nearest neighbour natively inside SQLite, no JS loop:
```ts
export async function retrieve(query: string, k = 3) {
const [queryVec] = await embed([query])
const { rows } = db.execute(
`SELECT content, source, distance
FROM chunks
WHERE embedding MATCH ? AND k = ?
ORDER BY distance`, // smaller distance = more similar
[JSON.stringify(queryVec), k],
)
return rows?._array ?? []
}
```
Then **feed the retrieved chunks to the model**, the same `qwen` we loaded at the start. That 469 MB is a hard ceiling: the phone has ~900 MB of usable RAM and the 119 MB embedder is already resident. I shipped Qwen2.5-1.5B (1.07 GB) first; it loaded fine in the simulator, then hard-crashed the phone the moment both models were in memory, an uncatchable native OOM, no JS error to trap, just a dead process. The 0.5B fits alongside the embedder with room to spare.
Generation is a single `completion()` call, with the retrieved chunks as the context:
```ts
const SYSTEM_PROMPT =
'Answer using only the provided context. Each context line starts with the name of ' +
'whoever sent it, before the colon. Never name a person who does not appear in the ' +
'context. Reply with one short sentence. If the answer is not in the context, say you do not know.'
const question = 'Who ended up paying for the hotel?'
const chunks = await retrieve(question)
const context = chunks
.map((c, i) => `[${i + 1}] (${c.source})\n${c.content}`)
.join('\n\n')
const senders = ['Ritesh', 'Ayush'] // the chat's participants, captured at import
const res = await qwen.completion({
messages: [
{ role: 'system', content: SYSTEM_PROMPT },
{ role: 'user', content: `Context:\n${context}\n\nQuestion: ${question}` },
],
n_predict: 96, // cap the answer length
temperature: 0, // greedy: this is extraction, not writing
stop: ['<|im_end|>', ...senders.map((s) => `\n${s}:`)],
})
const answer = (res.text ?? '').trim()
```
Those last two config lines, plus one line of the system prompt, earn their keep. `temperature: 0` because this is extraction, not writing: at `0.2` the same question gave a different answer on every rerun. The `stop` list keeps a small model from sailing past its answer and *continuing the chat log* in your friends' names. And the names rule in `SYSTEM_PROMPT` stops it from attributing a message to whoever else happens to be in the context.
## It works!
With the messages chunked and embedded into that vector DB, retrieval already answers a standalone question. Ask *"Who ended up paying for the hotel?"* and it replies **Ritesh ended up paying for the cottage**. Look at that answer again: we said "hotel", the chat only ever said *"cottage"* and *"the room on my card"*, and retrieval still landed on the right messages. That's the match on meaning we were promised. And the same question that overflowed the context earlier now costs a few hundred tokens instead of 21,000.
### ...until you ask a follow-up :(
Real conversations aren't standalone. The next thing you ask is a follow-up:
> How much was it?
Answer that on its own (embed *"How much was it?"*, search, generate) and the model has no idea what "it" refers to. The query points at nothing in particular (no hotel, no room, no trip), and a months-long chat is full of amounts: bus tickets, lunch splits, a phone battery. Retrieval drifts to whatever chunk sounds most like the question, and the model, handed a note about the wrong thing, gives up:
The nearest thing to *"How much was it?"* in the whole chat is a verbatim *"how much was it?"* from an unrelated conversation about a phone battery. The retriever hands the model that note; the model, told to answer only from context, can't connect a battery repair to anything and shrugs. On other runs it happily quoted the battery price instead. Either way, the follow-up is lost. The model isn't broken; it just never saw the messages that would have answered it.
> Always give the model an explicit way out in the system prompt ("if the answer is not in the context, say you don't know"): a small model *will* answer something, and without the escape hatch that something is a confident wrong guess instead of an honest shrug.
The fix is to carry the conversation forward, in two small changes: let retrieval inherit the previous question, and replay the recent turns to the model so it can resolve "it".
```ts
// 1. retrieval inherits the previous question, so "hotel" rides along
const prev = [...history].reverse().find((t) => t.role === 'user')
const chunks = await retrieve(prev ? `${prev.content}\n${question}` : question)
// 2. replay the recent turns so the model knows what "it" is
await qwen.completion({
messages: [
{ role: 'system', content: SYSTEM_PROMPT },
...history.slice(-6),
{ role: 'user', content: `Context:\n${context}\n\nQuestion: ${question}` },
],
// ...same n_predict / temperature / stop as before
})
```
Same question, same vector DB, now grounded:
The only thing that changed is that the follow-up remembered what came before.
> Replaying history fixes the follow-up, but it doesn't scale forever: that `history.slice(-6)` caps it at six turns for a reason. Replay *everything* and after enough back-and-forth the history alone overflows the 4,096-token window: the exact problem we started with, one level up. The usual answer is **compact + recent**: keep the last few turns verbatim, and fold everything older into a short running summary that the model refreshes as the chat grows. Recent turns stay sharp; old ones shrink to a sentence.
## Vector search beyond LLMs
Step back and look at what we just built, because the interesting half isn't the half that talks. The generator was the chatbot's shakiest link: we spent a system prompt, a stop list, and `temperature: 0` just keeping it honest. Everything that actually *found* the answer - chunk, embed, KNN - never needed a language model at all. Vector search lives mostly in AI apps, but the technique stands alone. Anything you can frame as "find the closest in meaning" fits:
- **Recommendations:** items whose vectors sit near ones a user liked.
- **Deduplication and plagiarism:** near-identical documents give near-identical vectors.
- **Anomaly detection:** the outliers sit unreasonably far from everything else.
So let's put that claim to the test: drop the LLM entirely and keep everything else. The same nitro-sqlite machinery and the same KNN, pointed at a photo gallery instead of a chat log - one you can search by *typing*, by *tapping*, or by *pointing your camera at something*. Fully offline, same as before.
## Building on-device image search
If the chatbot's machinery really carries over untouched, only one piece stands in the way: the embedder. MiniLM only reads text, so it has nothing to say about a photo: an image and the word "dog" never land in the same space, so there's no distance to measure.
Swap MiniLM for [**CLIP**](https://huggingface.co/openai/clip-vit-base-patch32), which embeds **images and text into one shared space**. (This is the one place the chatbot's single-runtime claim bends: CLIP ships as ONNX rather than GGUF, so this model runs on ONNX Runtime instead of llama.cpp.) A golden retriever photo and the words "a dog" land next to each other, so one table of image vectors buys three searches:
- **text โ image:** type "snowy mountains", embed it, KNN โ matching photos.
- **image โ image:** tap a photo, reuse its vector, KNN โ similar photos.
- **camera โ image:** shoot a photo (or upload one), embed it on-device, KNN โ the gallery photos that look like it.
Let's build a gallery of 1000 photos that does all three, starting with the one table that has to serve all of them.
One table, three searches
Same `vec0` table, just wider (CLIP is 512 dimensions), still cosine, the similarity CLIP was trained against. `react-native-nitro-sqlite-vec` ships typed helpers, so there's no SQL to hand-write for the table or the search:
```ts
import { createVectorTable, knnSearch } from 'react-native-nitro-sqlite-vec'
createVectorTable(db, 'images', {
dimensions: 512, // CLIP ViT-B/32 output size
distanceMetric: 'cosine',
column: 'embedding',
})
```
Seeding the gallery
Every image needs its vector. You *can* compute them on the phone with CLIP's **vision** tower (that's the camera, below), but for a gallery that ships with the app it's nicer to precompute them offline with the *same* weights and bundle them. 1000 ร 512 floats is 2 MB of raw little-endian float32, which we ship base64-encoded and insert in one batch:
```ts
db.executeBatch(
galleryVectors.map((vec, rowid) => ({
query: 'INSERT INTO images(rowid, embedding) VALUES (?, ?)',
params: [rowid, JSON.stringify(vec)],
})),
)
```
On a low-end phone (a 4 GB moto g35) that seeds all 1000 in **~1.1 s**, paid once at first launch, with an array mapping each `rowid โ image file`.
Search by typing (text โ image)
Now CLIP's **text** tower, loaded through [`react-native-nitro-onnxruntime`](https://www.npmjs.com/package/react-native-nitro-onnxruntime), a Nitro Module for talking to native ONNX Runtime built by [Ronald](https://x.com/BubbleTroubl_rg) (a margelo avenger!). Tokenize with CLIP's BPE vocab, embed the query, KNN:
```ts
const queryVec = await embedText('snowy mountains') // clip_text.onnx -> 512 floats
const hits = knnSearch(db, 'images', Array.from(queryVec), 48) // [{ rowid, distance }]
```
The text tower encodes in **~215 ms**, most of which is BPE tokenization and first-run session warmup rather than the network itself (the vision tower below clocks in cheaper); KNN over 1000 vectors comes back in **~8 ms**. Nobody typed the word "snow" into a caption anywhere: the match is purely geometric, and it only works because both towers came from the same CLIP model, sharing one space.
Search by tapping (image โ image)
No model call at all. The tapped photo's vector is already in the table, so "find similar" is a KNN with a vector we already have. **~5 ms**, faster than the tap animation:
```ts
// Ask for k+1: hit #0 is always the tapped photo itself, at distance 0.
const hits = knnSearch(db, 'images', Array.from(getEmbedding(rowid)), 49)
const similar = hits.filter((h) => h.rowid !== rowid)
```
Point the camera at it
The fun part: search by what the camera sees. Typing made its vector from text, and tapping reused one already sitting in the table. The camera is the first time we have to *make* an image vector on the device, which means running CLIP's vision tower on the phone, and the honest cost isn't the neural network, it's everything around it.
The obvious way to feed a photo to an embedder is to base64 it and pass the string to JS. But a camera frame is several megabytes, and encoding it, copying it across the bridge, then decoding it again wastes time and memory on a phone that has little of either. So [VisionCamera v5](https://visioncamera.margelo.com) captures to a file and hands back a filesystem path. That path never becomes a base64 string: it goes straight to native code.
```ts
const photo = await photoOutput.capturePhotoToFile({}, {})
const { vector } = await embedImageFile(photo.filePath)
```
And a function named `embedImageFile` is where the work happens:
```ts
const image = await loadImage({ filePath: path }) // platform JPEG decoder
const side = Math.min(image.width, image.height)
const square = await image.cropAsync( // center crop
(image.width - side) / 2, (image.height - side) / 2, side, side)
const small = await square.resizeAsync(224, 224) // CLIP's input size
const raw = await small.toRawPixelDataAsync() // { buffer, width, height, pixelFormat }
const pixels = ClipOps.clipNormalize(raw.buffer, raw.width, raw.height, raw.pixelFormat)
const { image_embeds } = await session.runAsync({ pixel_values: pixels })
```
[`react-native-nitro-image`](https://github.com/mrousavy/react-native-nitro-image) does the decode, crop and resize with the **platform's own** decoder, and hands back an `ArrayBuffer` of raw pixels. That matters for more than speed: it respects EXIF orientation (a portrait photo would otherwise reach CLIP lying on its side) and it reads HEIC, the default on every modern iPhone, which a bundled JPEG decoder simply cannot open.
[Video: Point the phone at a cat, shoot, and the gallery comes back sorted by cat-ness: a cat's nose, then bulldogs, labradors, a lion, a tiger, a husky. Recorded on a moto g35 in airplane mode.](https://margelo.com/videos/rag-camera-search.mp4)
As the clip shows, the gallery reorders itself around the concept. The shot lands in the same 512-d space as the rest, so it's **immediately** findable by text search too. The count ticks up to 1001.
> On the moto g35, the full shutter-to-results path is **decode 16 ms ยท CLIP 64 ms ยท KNN 5 ms**. The neural network is the *cheap* part. The expensive thing we removed was decoding a 12 MP sensor JPEG only to throw away everything outside a 224ร224 crop. Capping the capture at 1024ร768 cut the decode from 222 ms to 16 ms. Look at your pixels before you look at your model.
Or upload one
The camera and the photo library are the same code path underneath: a picked photo is just another file path, so **Upload a photo** is `launchImageLibrary()` feeding the exact same `embedImageFile()`. That makes it a good way to see what CLIP is really doing.
Here's a photo of some cats on a windowsill, dropped into a gallery of 1000 stock photos that contains, as far as I can tell, exactly one cat:
The nearest neighbour is a close-up of a cat's nose. Then it runs out of cats and falls back to bulldogs, black labs, pugs, foxes, pets and small mammals, in descending order of cat-ness. Nothing was tagged "cat", no caption searched: CLIP put the photo somewhere in 512 dimensions, sqlite-vec returned what lives nearby, and "nearby" meant *animals with faces*. When it can't find the thing, vector search degrades to the closest concept it has instead of returning nothing.
> The last mile is C++: reshaping raw pixels into CLIP's normalized CHW tensor is 150k float operations per photo, and the bundled vectors arrive as a 2.7 MB base64 blob. Both are pure buffer-walking, which is what a JS engine is worst at, so both live in a small [Nitro Module](https://nitro.margelo.com): `ArrayBuffer` in, `ArrayBuffer` out, straight over JSI.
>
> One trap in there. One platform hands back **BGRA** pixels, the other **RGBA**, so read `pixelFormat`, never assume. Guess wrong and you silently swap red and blue: the model still returns a confident vector, the app still works, and every result is quietly wrong. No crash, just bad neighbours.
## Where next
One direction worth exploring from here, and then some parting thoughts.
GraphRAG
Vector search finds the nearest chunk, but some questions are about **connections**. *"Which trips did we plan but never take?"* spans people, events, and several threads, so no single chunk holds the answer. **GraphRAG** extracts the entities and relationships in your data into a knowledge graph and answers by *walking it*. Microsoft's [*From Local to Global*](https://arxiv.org/abs/2404.16130) paper introduced the technique, and LinkedIn's [knowledge graph RAG](https://arxiv.org/abs/2404.17723) beat conventional retrieval by **77.6% in MRR**.
Photos give you that graph for free: GPS is a *place*, the timestamp a *day*, face recognition a *person*. In SQLite that is two more tables next to the `vec0` one, and *"photos of Ayush at the beach last summer"* becomes a graph walk with vector search handling the fuzzy "beach" part. Still one file, still offline. That build deserves its own post.
Final thoughts
A WhatsApp chatbot and a photo gallery are the *same* kind of `vec0` table and the *same* KNN, differing only in the embedding model. Once your data is vectors, "search by meaning" becomes a primitive you can point at almost anything, in a SQLite file you already ship, on a $100 phone in airplane mode.
Everything built in this post, the WhatsApp import, chunking, embeddings, `vec0` search, lives in one repo you can read end to end. It's built on top of the AI Chat App Demo from [the "Building a Native LLM Chat App" blog post](https://margelo.com/blog/building-native-llm-chat-app-with-rag):
[On-device WhatsApp RAG: nitro-sqlite-vec + local Qwen](https://github.com/margelo/ai-chat-demo-with-rag)
The one loud failure was `Context is full`. Most of the rest are silent: a chunker that drops a name, an English-only embedder that mangles a bilingual chat, two byte orders that swap red and blue. All fluent, confident, wrong, and error-free. On-device RAG fails quietly, so suspect the data path long before the model.
> Want fast AI running entirely on the phone (RAG, semantic search, local inference) at native speed in your React Native app? Reach out and we'll help you ship it.