Personal tools

Vision-Language-Action Models

University of Washington_091921A
[University of Washington]
 

- Overview

A vision-language-action (VLA) model is a multimodal AI model that takes visual inputs and text instructions to directly output physical or digital actions. Unlike traditional robotics pipelines that separate perception, planning, and control into distinct modules, VLAs unify reasoning and action generation within a single system.

1. How VLAs Work:

  • Vision Encoder: Processes camera feeds or screen frames to capture spatial geometry and object semantics.
  • Language & Reasoning Layer: Interprets natural language instructions and grounds them to visual context.
  • Action Decoder: Translates multimodal representations into executable commands. Common output methods include discrete action tokens, action chunks, diffusion policies, and flow matching.

 

2. Notable Models & Applications:

  • Robotics & Humanoids: Pioneered by systems like RT-2, followed by OpenVLA, Octo, π0, Helix, GR00T N1, and Gemini Robotics for manipulation, navigation, and whole-body control.
  • Virtual & Digital Agents: Applied beyond physical robots to areas like autonomous driving and real-time screen interaction, such as the Gamer foundation model for 3D game environments.

 

 

 

[More to come ...]


Document Actions