Multimodal Large Model Evaluation Tutorial: Image-Text Understanding, OCR, and Hallucination Checks

Multimodal large model evaluation is used to test the reliability of a model’s understanding of images, videos, and documents. By combining image+QA pairs, SimpleVQA, visual grounding, and hallucination checks, it helps teams design reusable evaluation sets and support model selection.

What Is Multimodal Large Model Evaluation?

Multimodal large model evaluation uses structured tasks and data to test whether a model truly understands multimodal inputs such as images, videos, documents, and web pages, and whether it can provide accurate, verifiable answers. It does not only look at whether the answer is fluent; it also focuses on whether the answer conforms to visual evidence, whether the model can recognize text and objects, whether it can make localization judgments, and whether hallucinations are present.

Multimodal large models usually provide visual understanding capabilities for images, videos, and multimodal inputs. Their capability range can cover tasks such as image captioning, visual question answering, object recognition, 3D localization, document and web page parsing, long-video understanding, and OCR text extraction. Therefore, evaluation also needs to expand from a single question-answering task into a system that covers different inputs, different tasks, and different types of errors.

Key point: The core question in multimodal large model evaluation is not “can the model speak,” but “does the model speak correctly based on the input content.”

Evaluation focusQuestion to answerCommon tasks
Image-text understandingWhether the model understands image or video contentImage captioning, visual question answering, factual question answering
OCR capabilityWhether the model can read and understand text in images, documents, or web pagesOCR text extraction, document parsing, web page parsing
Visual groundingWhether the model can map answers to real targets or spatial positionsObject recognition, position judgment, 3D localization
Hallucination controlWhether the model fabricates information that does not exist in the inputFactual accuracy checks, evidence consistency checks

How to Break Evaluation Tasks into Image-Text Understanding, OCR, Visual Grounding, and Hallucination Checks

Multimodal evaluation should not rely on only one overall score. A more robust approach is to first break it down by task, and then design samples, expected answers, and error types for each type of task.

Image-Text Understanding Evaluation

Image-text understanding evaluation focuses on whether the model can answer natural-language questions based on images or videos. It can be implemented through visual question answering and image captioning tasks, such as asking the model to describe a scene, identify main objects, and answer short-answer questions related to the visual content.

This type of task is suitable for observing two capabilities:

  • Whether the model can extract key information from images or videos;
  • Whether the model’s answer remains factually consistent with the input content.

If the model’s answer seems reasonable but is inconsistent with the image content, it should be recorded as a factual error or hallucination risk, rather than simply being considered good language expression.

OCR Evaluation

OCR evaluation focuses on the model’s ability to extract text from images, documents, or web pages, and to understand the layout and semantics of the text. It is suitable for input scenarios such as receipts, screenshots, tables, web pages, and instructional documents.

When evidence is insufficient, specific OCR metric formulas or thresholds should not be introduced arbitrarily. A more general evaluation design can start with the following questions:

  • Whether the text is read completely;
  • Whether key fields are missed or misread;
  • Whether the model can understand field meanings by combining layout relationships;
  • Whether the model fabricates content when the image contains no relevant text.

Visual Grounding Evaluation

Visual grounding evaluation focuses on whether the model can recognize objects and provide reliable judgments related to position. Questions can be designed around object recognition, spatial relationships, and 3D localization, such as asking whether a certain target exists, where the target is located in the image, and whether a spatial relationship exists between two objects.

The focus of this type of evaluation is not to make the model generate longer explanations, but to observe whether the answer can correspond to a real target in the image. If the model identifies an object that does not exist, or states the position relationship in reverse, it should be recorded as a localization deviation or a hallucination-related error.

Hallucination Checks

Multimodal hallucination refers to answers given by the model that are not supported by the input content, such as fabricating objects that do not exist in an image, reading text that does not exist, pointing to the wrong target in a localization question, or giving a definitive conclusion when evidence is insufficient.

Hallucination checks can use factual accuracy as the core observation point and combine the following signals:

  • Whether the answer is consistent with the evidence in the image, video, or document;
  • Whether objects, text, or relationships that do not exist in the input appear;
  • Whether the model is overconfident in uncertain scenarios;
  • Whether human review results support the automatic scoring conclusion.

How Evaluation Data Should Be Organized into Images and Question-Answer Pairs

Image+QA pairs are a common data organization format in multimodal large model evaluation. They bind visual inputs with natural-language questions, making it easier to check whether the model answers based on the image content, and also making it easier to cover tasks such as recognition, understanding, reasoning, OCR, and factual question answering.

A basic sample usually contains three parts:

FieldRoleExample description
Image or multimodal inputProvides visual evidenceImages, document screenshots, web page screenshots, video clips, etc.
QuestionClarifies the evaluation targetAsking about objects, text, positions, facts, or relationships
Answer or judgment criteriaUsed for scoring and reviewStandard answers, acceptable answer ranges, error types

Construct Samples in Layers by Task Type

To avoid an evaluation set that only covers simple recognition, samples can be designed in layers according to task type:

Task layerSample objectiveMain observation points
Image captioningDescribe the overall scene and key objectsWhether key information is missed, whether objects are fabricated
Visual question answeringAnswer questions related to image contentWhether the answer conforms to visual evidence
OCR extractionRead text in images or documentsWhether text is misread, missed, or fabricated
Localization judgmentJudge object positions or spatial relationshipsWhether it corresponds to real targets and positions
Document parsingUnderstand structured information in documents, web pages, or screenshotsWhether field meanings and context are understood
Long-video understandingAnswer questions based on video contentWhether key clips are omitted or temporal information is confused

Control Sample Size and Ensure Question Quality

An evaluation set is not necessarily better simply because it is larger. For model evaluations that require rapid iteration, the data can first be kept at a scale that is convenient for human review and repeated runs; in some development documents, there are also recommendations to keep the amount of multimodal evaluation data below 1000 items.

What matters more is question quality. High-quality questions should cover multiple tasks and scenarios and be somewhat challenging, rather than only testing simple recognition such as “what is in the image.” For business evaluations, high-frequency samples and high-risk samples from real usage scenarios should also be included.

Recommendation: A small-scale evaluation set can also be valuable, as long as the samples cover key tasks, the questions are sufficiently clear, and each sample has a reviewable expected answer or judgment criterion.

What Capabilities Is SimpleVQA Suitable for Evaluating?

SimpleVQA is mainly used to evaluate the factual accuracy of multimodal large models when answering natural-language short-answer questions. It covers multiple tasks and scenarios, emphasizes question quality and challenge, and is suitable for inclusion in the basic factual accuracy evaluation stage.

In multimodal large model evaluation, the ideas behind SimpleVQA can be used to check whether the model answers questions based on image content and whether its answers are consistent with the facts. It is especially suitable for image-text question answering, short-answer fact verification, and evaluation of basic understanding capabilities.

However, SimpleVQA should not be treated as a complete evaluation system. For practical model selection, it is also necessary to add evaluations for OCR, visual grounding, long-video understanding, document parsing, and business samples.

UsageSuitable for evaluatingShould not replace
Natural-language short-answer questionsFactual accuracy, basic image-text question-answering capabilityDedicated OCR evaluation
Multi-task scenario questionsAnswer reliability across different scenariosDedicated visual grounding evaluation
Challenging questionsWhether the model truly understands the inputBusiness-customized evaluation

How to Combine Visual Grounding and Hallucination Metrics for Judgment

Visual grounding and hallucination control are not two independent issues. Many multimodal hallucinations are related to unreliable localization: the model may identify the wrong target, or it may treat an object that does not exist in the image as a fact when answering.

Observe Hallucination Risk from Localization Questions

Visual grounding questions can be designed around object recognition, spatial relationships, and 3D localization. During evaluation, the focus should be on whether the answer can correspond to a real target in the image, rather than only on whether the language is smooth.

Common errors can be recorded as:

  • The target does not exist, but the model confirms that it exists;
  • The target exists, but the position judgment is wrong;
  • The spatial relationship is stated in reverse or oversimplified;
  • The model gives a definitive description when evidence is insufficient.

Observe Jointly Using Factual Accuracy, Human Preferences, and Automatic Metrics

Hallucination risk can be observed jointly through factual accuracy, consistency between the answer and image evidence, human preference judgments, and automatic metrics. Related research also includes approaches that improve visual grounding and reduce hallucinations through fine-grained reward models, and update pretrained multimodal language models by combining human preferences and automatic metrics.

This shows that there is a connection between grounding capability and hallucination control: if the model can more reliably anchor its answers to visual evidence, it is more likely to reduce unsupported generation.

Note: Automatic scores can improve evaluation efficiency, but they should not be used alone as the conclusion. For high-risk samples, boundary samples, and business-critical samples, human review or preference annotation should be retained.

Evaluation Checklist for Multimodal Model Selection

Multimodal model selection cannot look only at a single accuracy number. Choices in real scenarios usually also depend on factors such as task type, response speed, GPU memory budget, Chinese-language performance, and OCR capability.

Design an Evaluation Table by Task Weight

Different businesses assign different weights to capabilities. Document understanding scenarios may place greater emphasis on OCR and layout semantics; video analysis scenarios place more emphasis on long-video understanding; and spatial perception scenarios may focus more on object recognition and localization judgment.

User needSolutionEffect
Determine whether image content is correctly understoodSet image captioning and visual question-answering samplesDiscover factual errors and missing information
Read text in screenshots, documents, or web pagesSet OCR extraction and document parsing samplesDiscover misread text, missed text, and field confusion
Judge target positions or spatial relationshipsSet object recognition, position judgment, and 3D localization questionsDiscover localization deviations and target confusion
Reduce unsupported answersSet conflicting samples, missing-evidence samples, and human reviewIdentify hallucinations and overconfident answers
Support engineering selectionRecord response speed, GPU memory budget, Chinese-language performance, and OCR capabilityAvoid selecting models based only on a single score

To make evaluation reusable, it is recommended to record error types for each type of sample, rather than only recording right or wrong. Common errors include:

  • Hallucination: fabricating objects, text, or facts that do not exist;
  • Factual error: the answer is inconsistent with evidence in the image, video, or document;
  • Missed text: key fields are omitted in OCR scenarios;
  • Misread text: similar characters, numbers, or fields are read incorrectly;
  • Localization deviation: the recognized target is wrong or the position relationship is wrong;
  • Long-video omission: key clips are ignored or chronological order is confused;
  • Chinese-language expression issue: Chinese understanding, generation, or terminology expression is unstable.

A Reusable Evaluation Process

Clarify Task Boundaries

First determine the main scenarios the model will serve, such as image-text question answering, document parsing, OCR extraction, visual grounding, or long-video understanding. The clearer the task boundaries are, the easier it is to design evaluation samples.

Construct Image+QA Pairs

Prepare images, document screenshots, or video clips around key tasks, and design natural-language questions and expected answers for each input. Questions should cover simple recognition, complex understanding, fact verification, and high-risk boundary cases.

Record Results in Layers

Do not record only an overall score. Results should be recorded by dimensions such as image-text understanding, OCR, localization, hallucination, Chinese-language performance, and response speed, making it easier to identify the model’s strengths and weaknesses.

Add Human Review

For hallucination, localization, and business-critical samples, it is recommended to add human review. Human judgment can supplement semantic details and contextual judgments that are difficult for automatic metrics to cover.

Repeatedly Compare Using the Same Evaluation Set

When the model version, prompt, or deployment conditions change, rerun using the same evaluation set. This makes it possible to compare the performance of different models or different configurations more stably.

Conclusion: Multimodal large model evaluation should jointly consider task decomposition, data organization, factual accuracy, localization reliability, and hallucination control. Only by combining automatic evaluation with human review can reliability judgments become closer to those in real business scenarios.