VISTA: A Multimodal Egocentric Benchmark
for Goal-Oriented Perception and Assistance
to Blind and Low-Vision Users

Noé Mosca^1,2,* Mohamed Rayan Barhdadi^3,* Muhammad Hamza^4,* Joshua Yeh^1,* Suryodaya Bikram Shahi^1,* Zhiyan Li¹ Selim-Antoine Lali¹ Weiyue Li¹ Hasan Kurban⁵ Erchin Serpedin³ Gang Luo¹ Yue Zhang^1,† Minghan Li^1,† Mengyu Wang^1,†

¹Harvard AI and Robotics Lab, Harvard University ²Télécom Paris, Institut Polytechnique de Paris ³Texas A&M University ⁴University of Toronto ⁵Hamad Bin Khalifa University

^*Contributed equally as co-first authors. ^†Contributed equally as co-senior authors.

PDF Dataset (Hugging Face) Code

Abstract

Blindness and low vision limit many people’s ability to complete daily tasks, motivating AI assistants that provide useful guidance. Yet, existing benchmarks do not adequately test whether vision models can provide effective everyday assistance. In particular, i) existing benchmarks primarily rely on visual and textual inputs, with limited support for other modalities that are increasingly available in egocentric wearable devices such as gaze, inertial measurements, spatial trajectories, and others; and ii) existing evaluations emphasize scene understanding and question answering, with limited coverage of goal-conditioned guidance for blind and low-vision users across everyday tasks. To bridge this gap, we introduce VISTA (Visually Impaired Scene and Task Annotations), a multimodal egocentric benchmark for goal-oriented assistance for blind and low-vision users. VISTA contains ∼1,000 egocentric recordings captured with Project Aria glasses; each recording provides synchronized sensor streams, six of which serve as model inputs, and three different types of annotation, covering ten task categories (TC), from object localization to hazard detection. We evaluate 14 representative vision-language models zero-shot on VISTA and design a lightweight multimodal model baseline that integrates all six modalities. Experimental results show that, although existing models can often describe egocentric scenes, they remain limited in generating goal-directed assistive guidance. VISTA thus provides a benchmark for evaluating multimodal models that could improve everyday accessibility.

Aria RGB, stereo SLAM and eye cameras, audio, IMU and environmental sensors

Sensor modalities of the Aria glasses. VISTA records synchronized RGB video, stereo SLAM and eye-tracking cameras, audio, IMU, barometer, and magnetometer.

Explore the VISTA Dataset

VISTA contains 997 egocentric clips across ten task categories, split into 497 training, 50 validation, and 450 test clips. Each clip includes scene descriptions, question–answer pairs, and goal–instruction pairs to support scene understanding, answering questions, and goal-oriented assistance. The three examples below illustrate object localization, hazard detection, and navigation; hover over a video to explore its recorded modalities or annotations.

Loading examples…

Zero-shot results on VISTA

Reported results use 8 uniformly sampled RGB frames plus task text, evaluated on 1,828 items from 450 test clips: 450 scene descriptions, 510 Q&A items, and 868 guidance items. Judge: GPT-6 Luna (gpt-6-luna), with high reasoning effort and the rubric below.

Judge scores (0–5). EG: evidence grounding (0–1.75); AC: answer correctness (0–1.75); TC: task completion (0–1); ACom: accessible communication (0–0.5). Scene, Q&A, and Guidance report the mean score per annotation type. Bold/underline: best/second-best per column within each model group; –: size not disclosed.

Model	Size	Score	Rubric components				Response type
Model	Size	Score	EG	AC	TC	ACom	Scene	Q&A	Guidance
Open-source models (frames + text)
InternVL3	1B	2.39	0.82	0.79	0.47	0.33	3.49	3.09	1.42
InternVL3	2B	2.75	0.96	0.91	0.54	0.35	3.77	3.19	1.96
InternVL3	8B	3.16	1.09	1.07	0.64	0.38	3.72	3.48	2.69
InternVL3	9B	3.13	1.08	1.06	0.63	0.38	3.63	3.32	2.76
InternVL3	14B	3.33	1.15	1.13	0.68	0.40	3.81	3.62	2.91
VideoLLaMA3	7B	1.76	0.61	0.57	0.32	0.27	2.48	2.43	1.00
Qwen3-VL	2B	2.65	0.91	0.89	0.54	0.33	3.48	3.33	1.82
Qwen3-VL	4B	3.22	1.11	1.10	0.68	0.38	3.78	3.69	2.65
Qwen3-VL	8B	3.31	1.15	1.13	0.69	0.39	3.76	3.69	2.85
Gemma 3	4B	2.73	0.92	0.91	0.58	0.34	3.19	3.49	2.04
Gemma 3	12B	3.07	1.04	1.05	0.66	0.36	3.41	3.76	2.48
Gemma 3	27B	3.06	1.03	1.05	0.66	0.35	3.32	3.81	2.48
Proprietary models (frames + text)
GPT-Realtime Mini	–	3.42	1.16	1.17	0.71	0.40	3.90	3.86	2.91
Gemini 3.1 Flash-Lite	–	3.75	1.27	1.30	0.80	0.42	3.92	4.20	3.39

Multimodal baseline

We test whether synchronized sensor streams improve answer quality by training language LoRA adapters and a sensor-fusion network on VISTA in two stages, using 8 then 28 sensor tokens. Answers are generated with greedy decoding and evaluated with the same judge and rubric as the zero-shot baselines.

Multimodal VISTA architecture: RGB frames pass through a frozen vision encoder; sensor streams pass through temporal adapters, reliability gates and bottleneck exchange before entering the VLM alongside text tokens. LoRA and sensor fusion are trainable. — **Multimodal baseline.** A frozen vision encoder processes RGB frames, while five sensor streams—IMU, magnetometer/barometer, eye gaze, hand tracking, and 6DoF trajectory—produce 8 or 28 sensor tokens through temporal adapters, question-conditioned reliability gates, and bottleneck exchange. Only the LoRA adapters and sensor-fusion network are trained.

VISTA versus pretrained backbones

Each backbone is compared before and after adaptation with VISTA’s sensor-fusion network and language LoRA adapters. Scores use the same judge and rubric as the zero-shot evaluation, with results reported overall, by rubric component, and for scene description, question answering, and guidance. Shaded rows show the adapted VISTA models; green percentages indicate the relative improvement in overall score over the corresponding pretrained backbone.

Model	Size	Method	Score	Rubric components				Response type
Model	Size	Method	Score	EG	AC	TC	ACom	Scene	Q&A	Guidance
InternVL3	2B	Pretrained	2.75	0.96	0.91	0.54	0.35	3.77	3.19	1.96
InternVL3	2B	VISTA (ours)	3.54 (+28.7%)	1.24	1.20	0.68	0.42	3.82	3.78	3.28
InternVL3	8B	Pretrained	3.16	1.09	1.07	0.64	0.38	3.72	3.48	2.69
InternVL3	8B	VISTA (ours)	3.72 (+17.7%)	1.29	1.27	0.73	0.44	4.03	4.07	3.39
Qwen3-VL	4B	Pretrained	3.22	1.11	1.10	0.68	0.38	3.78	3.69	2.65
Qwen3-VL	4B	VISTA (ours)	3.95 (+22.7%)	1.39	1.36	0.76	0.44	4.25	4.18	3.69
Qwen3-VL	8B	Pretrained	3.31	1.15	1.13	0.69	0.39	3.76	3.69	2.85
Qwen3-VL	8B	VISTA (ours)	3.93 (+18.7%)	1.39	1.35	0.76	0.44	4.19	4.04	3.74
Gemma 3	12B	Pretrained	3.07	1.04	1.05	0.66	0.36	3.41	3.76	2.48
Gemma 3	12B	VISTA (ours)	3.24 (+5.5%)	1.08	1.10	0.64	0.43	3.00	3.30	3.32

BibTeX

@misc{mosca2026vista,
    title   = {{VISTA}: A Multimodal Egocentric Benchmark for Goal-Oriented Perception and Assistance to Blind and Low-Vision Users},
    author  = {Noé Mosca and Mohamed Rayan Barhdadi and Muhammad Hamza and Joshua Yeh and Suryodaya Bikram Shahi and Zhiyan Li and Selim-Antoine Lali and Weiyue Li and Hasan Kurban and Erchin Serpedin and Gang Luo and Yue Zhang and Minghan Li and Mengyu Wang},
    year    = {2026},
}