Zina: Multimodal Fine-grained Hallucination Detection and Editing

Multimodal Large Language Models (MLLMs) often generate hallucinations, where the output deviates from the visual content. Given that these hallucinations can take diverse forms, detecting hallucinations at a fine-grained level is essential for comprehensive evaluation and analysis. To this end, we propose a novel task of multimodal fine-grained hallucination detection and editing for MLLMs. To tackle this task, we propose ZINA, a novel method that identifies hallucinated spans, classifies their error types into six categories, and suggests appropriate refinements. We trained ZINA on 20k synthetic samples generated via a new graph-based approach that captures the dependencies among errors. Moreover, we constructed the VisionHall dataset, which contains approximately 6.9k outputs generated by twelve MLLMs, with hallucinated spans manually annotated. We demonstrated that ZINA outperformed existing methods, including GPT-4o and LLama-3.2, in both detection and editing tasks.