Multimodal RAG: Build for Images and Tables
Written By:
Matt Tanner
Published on
A table’s headings can matter as much as its numbers. They tell a reader what each value measures, a relationship a chatbot also needs when answering questions from a manual.
Consider this example: a maintenance chatbot is asked for the allowed inlet pressure of pump P-4. The inlet is where fluid enters the pump. The example manual shows:
Pump | Inlet pressure | Outlet pressure |
|---|---|---|
P-4 | 2 bar | 8 bar |
The answer is 2 bar. Extracting only “P-4, 2, 8” loses the headings that explain why.
Multimodal RAG gives us a choice: preserve that relationship in structured text or retrieve the original image. The useful test is whether the retrieved evidence lets the chatbot choose the right value.
When text-only RAG loses the evidence
Multimodal retrieval-augmented generation retrieves external evidence across relevant modalities and supplies that evidence, or a useful representation of it, to a generator. For document question answering, that can mean searching captions, retrieving the associated page images, and asking a vision-language model to answer from those pages. The index and generator need not consume the same representation.
Uploading an image to a chatbot doesn't establish a retrieval pipeline. The distinguishing step is selecting evidence from an external collection before answering. Hugging Face's document RAG recipe makes the separation explicit: a document retriever selects pages, then a separate vision-language model receives the selected images and question.
Keep your text baseline. A parser that faithfully preserves the pump table's headers, rows, and units may already provide everything needed for that question. An image isn't inherently a better representation of a number.
Other questions depend on visual relationships: which connector an arrow points to, which plotted line belongs to a legend entry, or whether a chart's vertical axis uses percentages rather than absolute counts. These are useful test cases because a plausible summary can omit the detail needed to answer them. Use those cases to decide whether vision adds useful evidence.
This guide concerns document text, images, and tables. It doesn't assume that one multimodal RAG pipeline also accepts audio, video, or arbitrary combinations of modalities.
Choose the representation and retrieval path
Make two decisions separately: what makes evidence searchable, and what the generator actually receives. A caption can work well for discovery while remaining insufficient for an exact answer.
The following multimodal RAG architecture choices overlap. Use the table as a shortlist, not a progression from basic to advanced. Account for both ingestion work and per-query work.
Path | Evidence and retrieval unit | Ingestion and query work | Integration burden and likely failure |
|---|---|---|---|
OCR, captions or structured tables indexed as text | Text chunk or table row group; layout survives only if explicitly represented | Extract or describe content at ingestion; retrieve and generate over text | Reuse text retrieval; lost headers, units or image details can make an answer impossible |
Text retrieval linked to original images | Caption or passage finds a page/region; generator receives its original visual evidence | Caption/index work plus image loading and visual generation per query | Maintain reliable source links; a detail absent from the searchable text may never retrieve |
Visual retrieval or separate modality indexes | Page/region images, or candidates from text and visual indexes | Encode images; score/rerank candidates; assemble selected evidence | Match the retriever's scoring contract; inspect storage, fusion and small-detail failures |
These paths adapt the representation alternatives in NVIDIA's multimodal retrieval architecture discussion. Its separate-store approach retrieves candidates from each modality and reranks them; it doesn't require every modality to share one encoder.
Text representations. For the pump table, serialize a row as model=P-4; inlet_pressure_bar=2; outlet_pressure_bar=8, rather than P-4 2 8. Include the table title and applicable footnotes. A table split across pages needs its headers carried forward deliberately. Check that your parser keeps those relationships.
Unstructured's hi_res PDF strategy can extract image/table crops and table structure; its docs flag multi-column ordering limitations. Docling exposes layout, reading order, and table structure with lossless JSON export. PyMuPDF's Page.find_tables() and Table.extract() offer table-to-cell extraction when that narrower path is enough.
For image descriptions, decide what to include based on likely questions. “Pump assembly diagram” may support a topical search but not a query about a particular valve. Caption-linked images let the generator inspect more detail once retrieval succeeds; they don't repair a missed retrieval.
Compatible multimodal embeddings. A shared vector space requires encoders trained to produce compatible representations. CLIP, for example, has text and image encoders whose features are projected into a common space for similarity scoring. Storing unrelated text and image vectors with equal dimensions in one vector database does not establish that alignment. CLIP's default configuration uses 77 text positions and 224-pixel images, but checkpoint resolutions vary. Test small text and dense tables on the checkpoint you choose. See the CLIP model documentation.
Page-image retrieval. ColPali represents document pages using multiple vectors and compares them with text-query vectors through late interaction. It is a retriever, not an answer generator, and its scoring is not interchangeable with a single pooled-vector similarity search. The ColPali documentation describes the page-image representation and retrieval interface. The ColPali paper reports 257.5 KB per page for float16 embeddings; compression and index overhead affect the deployed footprint. Check that your backend supports its multivector late-interaction scoring.
If you combine separate indexes, retain each candidate's source identity and retrieval route. Use a deliberate fusion/reranking policy rather than treating scores from different models as interchangeable. For example, compare rank-based candidate fusion with a reranker on your labeled cases. Record the candidate budgets per path, then deduplicate before assembling context. That gives you a testable policy instead of an unexplained mixture of scores.
Build a document pipeline that preserves source identity
The vector store should help find evidence, not become its only surviving form. Keep a source-record store alongside derived representations. Our RAG pipeline guide covers the foundational stages; the extra requirement here is preserving the relationship between a searchable representation and its original evidence.
A source record can keep these fields:
Keep table cells or image bytes in the source record or in a resolvable asset store. Add a bounding box and coordinate convention when region citations matter. A page number alone doesn't specify which of several tables supplied the answer. The runnable example below creates its own one-page fixture.
A practical multimodal RAG pipeline follows this evidence chain:
Ingest and identify. Assign document versions and page/region identities before producing captions or crops. Preserve table headers, units, and footnotes with the relevant cells.
Represent and index. Create the chosen searchable representations, each pointing back to its source record. Record the parser, captioner, and embedding versions used.
Retrieve and resolve. Run the user query, retain ranked result IDs, then resolve them to the intended source versions. Collapse duplicate representations of the same evidence without discarding distinct necessary regions.
Assemble and answer. Supply selected text, tables, or images in the generator's supported format, with explicit evidence labels. Ask for references to those labels and validate the returned IDs.
Searchable representations help find evidence; source IDs resolve the selected originals before generation. Choose or test text and visual retrieval paths, then supply selected text, tables, or images with evidence labels.
Define access, update, and deletion tests alongside that chain. Recheck permission at evidence resolution, remove obsolete derived records when their source changes, and test that old captions cannot pull a replacement document under an unchanged identifier.
Extract and retrieve a table and its source image
Start with a text-assisted baseline you can inspect. This example creates a small PDF containing the pump table and a diagram with a caption, then reopens it for extraction. It searches the extracted table text and caption and returns the original PDF or image. No vision model, GPU, or API key is required.
For this local example, you need:
Python 3.10 or later.
A new folder and virtual environment.
PyMuPDF 1.26.6, pinned so extraction matches the output below.
No GPU or API key.
Create and activate the environment, then install the parser:
On Windows, activate with .venv\Scripts\activate instead. Save the three code blocks below in one file named evidence_check.py, in the order shown. The script creates its own assets in evidence-demo.
Keep the 1.26.6 pin for the output below. Current PyMuPDF 2.x uses layout analysis inside find_tables() by default.
When porting to 2.x, use find_tables(use_layout=False) for pure line-based detection and recheck the extracted cells; do not add that newer argument to 1.26.6.
The default lines strategy suits this fixture because the script draws a grid. For borderless tables, test strategy="text" and its word-count thresholds.
Create the sample manual
The first block draws a pump table and a small diagram, then saves them as a single PDF.
Extract the evidence
Next, reopen that PDF and extract its table cells, printed caption, and embedded image. Each searchable record points back to the original PDF or extracted image.
Retrieve a source for each question
Finally, rank the two records by the words they share with each question. The selected record supplies either the table row or the original image.
From the same folder, run:
PyMuPDF may first print “Consider using the pymupdf_layout package for a greatly improved page layout analysis.” This is informational. The results should then be:
The one-page test PDF that the script generates: a ruled pump table, the Figure 1 caption, and the embedded diagram it extracts.
Open evidence-demo/manual-v2.pdf and evidence-demo/p1-image.png to inspect the sources. The extracted image should be 300 × 100 pixels: two boxes labeled P-4 and V-2 joined by a line.
The first query ranks the table first and returns 2 from the column headed “inlet bar”. The second ranks the extracted caption first and resolves the embedded diagram.
Ranking here counts keyword overlap with extracted table text or the printed figure caption. The diagram is found through that caption; its pixels never enter the score.
Our multimodal_processor can instead use a vision-capable model to write searchable image summaries. That brings image content into the index indirectly, but the summary may still omit details.
For this one-page fixture, the only caption belongs to the only image. With real documents, associate captions, tables, and images by page regions, and test that mapping. Before adding a vision-language generator, confirm that its selected evidence includes the detail needed to answer. A question about a visual detail missing from the caption may need a richer description or visual retrieval.
Evaluate retrieval and visual reasoning separately
An answer can fail even when the right page was retrieved. Your evaluation should tell you whether to change ingestion, retrieval, context assembly or the generator.
For each test case, record the question, required document version, relevant page/region IDs, expected answer or rubric, and whether the answer is actually available. Preserve the original evidence for review, rather than grading a visual answer only against an automatically generated caption.
Start with these slices:
Slice | Example test | What to inspect |
|---|---|---|
OCR and dense tables | Read the correct inlet-pressure column | Extraction, header association, units |
Charts | Identify a series and report its value | Legend, axis scale, unit and acceptable reading tolerance |
Diagrams | Identify the component reached by an arrow | Spatial relationship, not just component names |
Mixed pages | Combine a table value with a nearby exception | Whether both evidence regions reached the generator |
Unreadable or missing evidence | Answer from a blurred label or absent specification | Appropriate uncertainty or refusal, not invented detail |
Version-sensitive questions | Use the current manual, not an older caption | Source resolution and citation correctness |
Score retrieval first. Declare the retrieval unit and cutoff. Region-level Recall@5 counts the distinct relevant regions among the first five results, divided by all labeled relevant regions.
For example, suppose a question needs table t1 and its footnote f1. The top five results are t1, t1, a P-5 paragraph, the diagram caption, and an unrelated table. Only one of the two relevant regions was found: Recall@5 = 1/2 = 50%. Counting t1 twice would wrongly report 100%; the footnote still never reached the generator.
Keep ranked order when calculating the cutoff. If your index retrieves whole pages, report page-level recall separately.
A question that needs both a chart and a footnote also deserves an “all required evidence present” check. Partial recall can coexist with an unanswerable context. For questions with no relevant evidence, keep them out of recall's zero-denominator calculation and score abstention separately.
Then score the answer. Use separate checks for factual correctness, support in the supplied evidence, and citation accuracy. For the pump example, verify the value, unit, and inlet/outlet distinction; a citation to the correct page doesn't excuse reading the wrong column. Where multiple passages legitimately support an answer, allow the supported alternatives rather than demanding one arbitrary ID.
Run a diagnostic with hand-selected correct evidence. If the answer is still wrong, improving search alone won't fix that case. If it becomes correct, inspect the retrieval and context-assembly trace. Keep this diagnostic separate from end-to-end results, because it removes a real source of failure.
Compare the text baseline, caption-linked images, and visual retrieval on the same development cases. Fix the generator when isolating retrieval changes, then test generator variants separately. Reserve a held-out set for the final comparison.
Have a human inspect numerical and visual disagreements with automated graders. A text-only judge cannot inspect an image it never receives; a caption-based judgment evaluates the caption's account of the evidence. Our LLM-as-a-judge guide covers rubric design and calibration.
Finally, track ingestion time, index size, retrieval/reranking latency, image-loading time, generation latency and provider usage. Keep one-time indexing work separate from per-query work. Report those costs beside slice quality; don't hide a table-reading regression inside an improved overall average.
Compare configurations with our platform
We help you compare configurations once you have defined the evidence contract and evaluation cases. Use our platform to run them under the same scoring protocol, rather than treating “multimodal” as a switch that settles the architecture.
Here is how the pump manual would move through our multimodal pipeline. An Unstructured-style loader separates text, images, and tables. The configured multimodal_processor summarizes those elements; the diagram needs a vision-capable generator. Our engine embeds and searches the summaries as page_content.
The originals remain available. With artifact_storage_cfg=None, our engine stores them locally and records paths in metadata such as image_source and table_source. Setting it to False keeps only the summaries.
Our engine requires Python 3.12 or later, unlike the standalone example above. The rapidfireai init --multimodal setup installs its image/table-processing dependencies.
At query time, preprocess_fn can use rag.storage_client.get_image_base64() to load the selected image or get_html() to load a table's HTML. That lets the generator inspect the original instead of relying only on its summary. Our MMDocIR notebook and custom preprocessing example show how to assemble these text/image content blocks. Keep evidence IDs alongside prompts and answers, and score the results with your own metric functions.
First, retrieve the same pages for both candidates. Give one candidate only the extracted text and the other the text plus the original image. Now you can see whether letting the generator inspect the image helps.
Test a different retrieval method separately, keeping the generator and evidence labels fixed. To use ColPali, you need to build and test an adapter that implements LangChain BaseRetriever. Pass that adapter through the RAG spec's retriever parameter. The interface accepts your adapter; it does not supply a built-in ColPali integration.
Keep a per-case record of retrieved IDs, assembled evidence IDs, the answer, cited IDs, metrics and timings.
For dataset-wide rates, return per-batch counts and divide their totals in accumulate_metrics_fn. For example, add correct answers and scored answers across batches, then divide. Without a custom accumulator, our engine sums each metric across batches, which would incorrectly add per-batch accuracy ratios.
Our Converge workflow adds an automation layer that stops, clones and launches configurations over the execution engine. Keep production monitoring in your existing tools, and keep the visual-evidence rubric under your control. Neither automated search nor a running score establishes that an answer read the right table cell.
Start with one evidence-preserving baseline
Take a representative document and write questions that require its table headers, chart units and diagram relationships. Preserve the originals, build one baseline, and inspect where information disappears before adding another model.
Start with our Colab setup for small models and datasets. It demonstrates the comparison workflow on a free Colab T4, rather than this multimodal pipeline.
For the full document path, the MMDocIR notebook parses PDFs with Unstructured, preserves original artifacts and compares generator and search settings. It needs OpenAI and Pinecone API keys; provider usage is billed separately. You can also discuss the workload with us.
Multimodal RAG FAQ
How does multimodal RAG differ from a multimodal LLM?
A multimodal LLM processes supported input types. Multimodal RAG adds retrieval from an external collection and supplies selected evidence to a generator. A model's ability to read an uploaded image doesn't tell you whether your system can find the right image.
Must all modalities share an embedding space?
No. You can retrieve textual representations and resolve their original images, or combine candidates from separate indexes. When you do use a shared vector space, use compatible encoders rather than mixing unrelated embeddings.
When is OCR enough?
When the extracted representation preserves everything required for your questions, including structure and units. Test that claim against the original documents; readable words alone don't establish correct table reconstruction.
How do you test image evidence?
Label the required pages or regions, check whether retrieval and assembly deliver them, then grade answers against the original images. Include cases where the evidence is unreadable or absent, and inspect whether citations support the specific claims made.