RAG benchmark: Lookio and a plain retrieval pipeline

On 450 deliberately difficult questions, Lookio answered 371 fully correctly and a plain text-only RAG pipeline answered 132, with up to two attempts per question. That is 82.4% versus 29.3%. We have published the questions, expected answers, complete candidate answers and timings so you can inspect what those numbers mean.

A small RAG pipeline is straightforward to build: split documents into chunks, embed them, retrieve a few passages and ask a language model to answer. It is a useful starting point. Whether that starting point can answer from a spreadsheet, read a scanned note or reconcile two policies is a separate question.

We wanted to measure that difference without changing the answer model between systems. This article explains the baseline, the results and their limits. The evaluation repository contains the evidence; the Arvenne dataset is a separate project you can use with your own system.

What we tested

These are stress tests, not everyday questions. DataBench asks for counts, filters and calculations over whole tables. HybridQA asks questions whose answer needs a table row and the text linked to it. Arvenne mixes scanned notes, spreadsheets and rules that contradict each other. The questions people usually ask a knowledge base, such as a return window or a setup step, are simpler than most of these. The benchmark shows how each system holds up when the question gets hard.

We evaluated 150 questions from each of three datasets, for 450 questions per system:

Lookio takes longer: a median of 9.1 seconds per question against 3.1 for the baseline, both attempts included when a second one ran. The time gap is smallest on DataBench, 5.7 seconds against 3.5, and largest on HybridQA, 12.0 against 3.1.

Only fully correct answers receive a point. Partial answers do not. The same reference-based judge evaluates both systems without seeing which system produced the candidate.

Each system has up to two attempts. A second attempt is generated only when the existing answer is not fully correct, without showing it the previous answer, reference or grading feedback. Success is selected offline against the reference. These are therefore two-attempt evaluation scores, not single-request production accuracy or proof that the product can identify the better answer without an answer key.

DataBench and HybridQA are evaluated subsets with bounded corpora, not complete official benchmark runs. I created Arvenne as well as Lookio, so it should not be read as an independent endorsement. Its 150 questions and references were frozen before running systems against them. Publishing each dataset separately makes that relationship visible rather than hiding it in an overall score.

What we mean by a plain RAG

The baseline uses a defined setup, not an intentionally vague “standard RAG”:

• Plain text extraction. PDF text layers, document text, spreadsheet rows and HTML text. No OCR or interpretation of embedded images and charts.
• Recursive chunks of 500 characters, with a requested overlap of 50 characters. The source name is prefixed to each chunk after splitting.
• text-embedding-3-small embeddings and the ten highest-scoring chunks by cosine similarity for the original question.
• One answer-generation call using only that context, with instructions to say when it does not know. No reranker, query rewrite or table-execution tool.

Both systems’ answer-generation chat calls use Claude Haiku 5.5, with reasoning disabled. The judge uses the same model. Lookio is a proprietary document question-answering system, evaluated in its Deep configuration using a local harness.

That shared model matters, but it does not equalize preprocessing capabilities or the number of internal calls. This is an end-to-end system comparison, not an experiment that isolates retrieval alone. An open-source pipeline with OCR, better retrieval or calculation tools would be a different baseline and deserves its own evaluation. This result does not establish that every custom RAG would score 29.3%.

The baseline specification publishes the exact prompts, settings and extracted-document manifest.

A simple question can depend on how the file was read

Arvenne question ARB-002 asks: “What is the battery capacity of the Pup?” The expected answer is 400 Wh. The selected plain-RAG answer says the provided context does not state that capacity. Lookio’s selected answer gives 400 Wh and identifies the autumn catalogue.

This is a small lookup, not a long reasoning puzzle. It illustrates why the quality of a document assistant begins before the final answer prompt. If the relevant fact is missing from the context the reader receives, a more confident response would not solve the problem. Saying “I do not know” is better than inventing it, but it still does not answer an answerable question.

You can inspect this candidate and every other one in the full answer files, rather than relying on a screenshot of one successful chat. The files include unsuccessful candidates too.

A benchmark is useful only within its boundaries

This is an exploratory evaluation we conducted ourselves, not an independent ranking or a preregistered study. Dataset selection and configuration followed earlier exploratory work. The judge is automated and comes from the same model family as the answer model; it can make mistakes. A reference match is not a guarantee that every extra statement in a response is grounded.

The repository includes per-attempt wall times and the sum of the attempts actually executed. Those timings come from the local evaluation harness, exclude indexing and grading, and do not represent production API latency. We do not turn them into a speed promise.

The test does not measure production availability, permission enforcement, every Lookio mode or every document workload. It provides inspectable evidence for the two configurations on these 450 questions. The methodology spells out the scoring, selection and timing rules. You can recompute the published scores without making a model call.

The retrieval script is only part of what you operate

A retrieval demo can be the right choice for a small, well-controlled corpus. A shared knowledge system also needs a way to manage what gets ingested, which assistant can read it, who can ask questions and how applications reach it. Those responsibilities do not disappear when the first prompt works.

Lookio provides that surrounding platform:

• Resources and assistants: manage files, text, web pages and websites; give an assistant selected resources, all resources or resources carrying chosen tags. Website imports have page rules and synchronization settings.
• A programmatic interface: use the REST API, TypeScript/JavaScript SDK or CLI from your own applications and automations.
• Agent connections: use MCP with OAuth or scoped API keys. Agents work within the permissions of their connection rather than gaining unrestricted access to the workspace.
• Team controls: admins manage members; Pro and Custom support per-member permissions and assignments to the assistants each person can use.
• External access: share selected assistants through their own MCP and HTTP interfaces, with public or key-protected access where supported, instead of handing out workspace-wide credentials.
• Widgets and oversight: put a chat bubble or embedded assistant on allowed websites, and inspect conversation, answer, indexing and activity logs.

For example, an application can ask an existing assistant through the API:

curl https://api.lookio.app/v1/assistants/ASSISTANT_ID/query \
  -H "Authorization: Bearer $LOOKIO_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Idempotency-Key: $(uuidgen)" \
  -d '{ "question": "How long do I have to return an item?", "mode": "flash" }'

That integration example is separate from the benchmark’s Deep configuration. The API returns the answer and its supporting citations. Your application decides what to do with them.

Inspect the released answers, try the Arvenne dataset with your own setup, then create a Lookio assistant on documents you actually use. The useful comparison is whether it can answer your questions and fit the permissions and interfaces your workflow needs.