PruneVid: Visual Token Pruning for Efficient Video Large Language Models

In this paper, we introduce PruneVid, a visual token pruning method designed to enhance the efficiency of multi-modal video understanding.

Large Language Models (LLMs) have shown promising performance in video tasks due to their extended capabilities in comprehending visual modalities. However, the substantial redundancy in video data presents significant computational challenges for LLMs.

To address this issue, we introduce a training-free method that 1) minimizes video redundancy by merging spatial-temporal tokens, and 2) leverages LLMs’ reasoning capabilities to selectively prune visual features relevant to question tokens, enhancing model efficiency. We validate our method across multiple video benchmarks, which demonstrate that PruneVid can prune over 80\% tokens while maintaining competitive performance combined with different model networks.

We begin by segmenting the video into different scenes and then decouple the video tokens into static and dynamic ones.

Next, we compress the static tokens along the temporal dimension and merge similar tokens in the spatial dimension to further reduce redundancy.

Afterward, by using the question-to-video attention weights learned from an intermediate layer, we determine which tokens should be pruned to improve efficiency.

PruneVid is more efficient and effective than other token pruning methods on PLLaVA, ST-LLM, and LLaVA-OneVision across multiple video benchmarks.

PruneVid achieves the most efficient balance by yielding the fastest TTFT speed up, the largest FLOPs reduction, the smallest memory footprint, and the highest accuracy, clearly underscoring the advantages of our token pruning approach over existing techniques.

In the figure below, we present a side-by-side comparison demonstrating how our model selects tokens guided by attention scores, highlighting the LLM’s strength in focusing on informative regions related to the questions.