V-JEPA 2: self-supervised video models enable understanding, forecasting and planning

Revolutionizing Machine Perception: V-JEPA 2 and the Leap Toward Intelligent Video Understanding. Imagine a world where machines can watch and understand videos almost like humans do, learning from endless hours of visual experience and a handful of real-world interactions. This is the vision brought to life by V-JEPA 2, a groundbreaking self-supervised video model that marks a radical step forward in how artificial intelligence grasps, predicts, and acts within our physical reality. At its core, V-JEPA 2 draws inspiration from the remarkable way humans learn: by observing, predicting outcomes, and planning actions without explicit instruction. The model is first trained on a staggering one million hours of internet videos, absorbing everything from the subtle motion of a hand to the complex choreography of daily life. During this phase, it isn’t told what to look for; instead, it learns to reconstruct missing pieces in video clips, focusing on the parts of a scene that matter most for making sense of what’s happening. The model’s architecture leverages a joint-embedding predictive approach. Instead of simply memorizing what it sees, V-JEPA 2 builds a rich internal representation of the world—capturing dynamic relationships, anticipating future events, and ignoring irrelevant details. This enables it to excel in classic vision tasks, such as motion understanding and object recognition, but also to tackle more sophisticated challenges like action anticipation and answering questions about videos with unprecedented accuracy. But the true magic happens when V-JEPA 2 is equipped for interaction. By exposing it to a small set of robot trajectories—just 62 hours of unlabeled footage—it learns how actions influence the world. This transformed version, V-JEPA 2-AC, doesn’t merely watch; it can plan and execute tasks using robotic arms, such as picking up and moving new objects in unfamiliar environments. Astonishingly, it does this without ever having seen those specific robots or tasks before, and without any task-specific training or reward signals. What sets V-JEPA 2 apart is its ability to generalize. By leveraging the vast diversity of internet videos, it acquires a deep understanding of physical dynamics and human activities. When aligned with a large language model, it becomes capable of answering intricate questions about video content—demonstrating not just rote recall, but genuine comprehension and reasoning about the temporal and causal structure of events. This approach moves beyond previous methods that relied heavily on expensive and limited real-world interaction data. V-JEPA 2’s massive pretraining on web-scale videos, followed by efficient adaptation using minimal robot data, opens the door to AI agents that can learn, predict, and plan in the real world with remarkable flexibility. In practical terms, it means robots could one day adapt to new tasks and environments as fluidly as people do, guided by a visual world model honed by watching the collective experience of humanity. V-JEPA 2 stands as a testament to the power of self-supervised learning and the promise of machines that can truly see, think, and act in our world. This is not just a step but a leap toward artificial intelligence capable of understanding and shaping the future as it unfolds before their eyes.