Can Visual Input Be Compressed? A Visual Token Compression Benchmark for Large Multimodal Models

Large multimodal models (LMMs) often suffer from severe inference inefficiency
due to the large number of visual tokens introduced by image encoders. While
recent token compression methods, such as pruning and merging, have shown
promise in reducing redundancy, their evaluation remains fragmented and incon
sistent. In this work, we present UniPruneBench, a unified and extensible bench
mark for visual token pruning in multimodal LLMs. UniPruneBench provides
standardized protocols across six ability dimensions and ten datasets, covering ten
representative compression algorithms and three families of LMMs (LLaVA-v1.5,
Intern-VL3, and Qwen2.5-VL). Beyond task accuracy, it incorporates system
level metrics such as runtime and prefilling latency to provide a holistic view.
Our experiments uncover several key findings: (1) random pruning is a surpris
ingly strong baseline, (2) no single method consistently outperforms others across
scenarios, (3) pruning sensitivity varies significantly across tasks, with OCR be
ing most vulnerable, and (4) pruning ratio is the dominant factor governing perfor
mance degradation. We believe UniPruneBench will serve as a reliable foundation
for future research on efficient multimodal modeling.