Meta-Video: Efficient Standardized Evaluation
for Video Understanding

Video understanding benchmarks are the primary yardstick for measuring progress in video-language modeling, and a growing body of work relies on them to claim improvements. The reported benchmark scores in these works, however, are almost always produced by frameworks (e.g., lmms_eval) used as black boxes: certain evaluation choices like frame sampling, preprocessing, and input formatting are inherited from preset defaults and seldom inspected carefully. We show these unexamined choices are a major source of variance. Even for the same model checkpoint, evaluation configuration alone can shift accuracy by up to 19.5pp. Hence, it is unclear whether reported gains reflect genuine improvements in capability or differences in evaluation setup. Moreover, many benchmark samples offer little evidence of visual capability, since they can be answered without the video, and the full suites are expensive to evaluate. We address these issues in two parts. First, we curate existing suites for discriminability and efficiency, producing Meta-Video, a balanced 1,000-question benchmark over 791 videos spanning diverse tasks and durations, which preserves model rankings, at 5× inference speedup. Second, we establish a standardized evaluation protocol by sweeping the configuration space to isolate choices that measurably affect results, distill them into best practices and reporting standards, and release reproducible reference scores across five models and seven benchmarks. We hope that the proposed evaluation framework and the Meta-Video benchmark will support video understanding research by providing a reproducible and efficient foundation to measure progress in a reliable way.

Recent papers report scores for the same Qwen2.5-VL-7B and Qwen3-VL-8B checkpoints that differ by far more than the 2–5pp gains those papers claim over the base model. The range of reported baseline scores per benchmark (Max Δ) across more than twenty recent video reasoning papers:

Several methods report gains over the Qwen2.5-VL-7B baseline that fall below our reproduction of the same baseline. When we re-evaluate three RL-based video reasoning models under identical settings, none of them improves consistently over the base model across frame counts:

Accuracy difference of Video-R1, VideoChat-R1.5 and Video-R2 to their Qwen2.5-VL-7B base model on Video-MME, under one fixed protocol, at 16 to 128 frames.

A reliable evaluation that supports fast model development therefore needs (1) a clean set of benchmark samples, (2) low evaluation cost while preserving rankings, and (3) a standardized evaluation protocol. Meta-Video addresses (1) and (2); the protocol and evaluation cards address (3).

We fix a reference configuration (bold in the figure at the top) and vary one factor at a time on Qwen2.5-VL-7B over Video-MME, TempCompass, MVBench, LongVideoBench and Meta-Video. What we found:

Best practice: evaluation cards

Inspired by dataset cards, every table in the paper comes with an evaluation card that records the full configuration behind its numbers: backend, batch size, number of frames, sampling strategy, decoder, max pixels, input modality, prompt and temperature. We recommend reporting these alongside any new video benchmark result.

Evaluating video models across multiple benchmarks and configurations is expensive, and questions that models answer without visual input inflate scores and obscure differences between models. Meta-Video is a compact benchmark of 1,000 question-answer pairs over 791 videos, selected from the evaluation sets of Video-MME, LongVideoBench, LVBench, VideoMMMU and VSI-Bench.

Construction. To filter questions answerable without visual input, we replace all frames with black frames and evaluate Qwen2.5-VL-7B-Instruct, Qwen3-VL-8B-Instruct and InternVL3.5-8B. Every question that all three models answer correctly is discarded. We then sample question-answer pairs from the remaining questions in each source benchmark, balanced across sources, video durations and question categories.

Question types. 937 questions are multiple-choice and 63 require a numerical free-form answer. 564 questions have four options, 185 five, and 170 more than five (157 with exactly ten). Correct answers are evenly spread across options: random guessing scores 20.3 and always picking the most frequent option scores 23.3.

Video duration. Videos run up to one hour: 335 short (<5 min), 281 medium (5–20 min) and 175 long (20–60 min). Benchmarks described as "long video" differ widely: videos average 41.1 min in the Video-MME long split, 7.9 min in LongVideoBench and 67.3 min in LVBench.

Source benchmarks and their categories.

Question types within each source benchmark.

Distribution of correct answer letters per benchmark.

Video duration distribution on Meta-Video.

Scores on Meta-Video are lower than on the source benchmarks by construction, since the questions that language priors solve are gone. Compare models by rank and by Meta-Video score, not against published numbers.

lmms_eval_video is a fork of lmms-eval with the vLLM video fix, an enforced frame cap, and Meta-Video in-tree as the task metav. It runs like any lmms-eval task:

git clone https://github.com/anymz1234/lmms_eval_video
cd lmms_eval_video && pip install -e .

python -m lmms_eval --model qwen2_5_vl \
    --model_args pretrained=Qwen/Qwen2.5-VL-7B-Instruct,max_num_frames=32 \
    --tasks metav --batch_size 1 --log_samples --output_path logs/metav_try

# Meta-Video score = macro average over the 5 sources
python eval_scripts/score_metav.py --write logs/metav_try/*/*_results.json

Each source is also a subtask (metav_videomme, metav_longvideobench_val_v, metav_lvbench, metav_video_mmmu_*, metav_vsibench) and re-uses its source task's loader, parser and metric, so every item is scored exactly as in its benchmark. The items live in lmms_eval/tasks/metav/data/: metav.jsonl (question, options, answer, video, duration, source ids, categories), metav_ids.json and metav_prompts.json.

The protocol sweeps are one command each:

bash eval_scripts/sweep_params.sh --model qwen25vl_7b --task videomme        # one knob at a time: backend, batch, decoder, sampling, frames, pixels, modality, temperature
bash eval_scripts/sweep_frames.sh --model qwen3vl_8b                           # accuracy vs 4..128 frames, .mp4 and .jpg
bash eval_scripts/sweep_vllm_fix.sh --model Qwen/Qwen2.5-VL-7B-Instruct --task metav   # HF vs stock vLLM vs fixed vLLM
python eval_scripts/collect_results.py --csv runs.csv

The code follows the lmms-eval license. The Meta-Video item lists and prompts are released under CC BY 4.0: http://creativecommons.org/licenses/by/4.0

Note: The videos belong to the source benchmarks (Video-MME, LongVideoBench, LVBench, VideoMMMU, VSI-Bench) and are subject to their own licensing terms. Please ensure compliance with the original dataset licenses.