Unraveling the World with AI: V-JEPA's Breakthrough

Imagine placing a glass of water on a table, concealing it behind a board, and then pushing the board towards where the glass should be. If the board passes through the glass's space, would an observer be astonished? Infants generally express surprise at such outcomes, developing an understanding of object permanence through observation. This intuitive grasp of physical reality is now within the reach of artificial intelligence systems.

Researchers have pioneered an AI-based system capable of learning about the environment solely through video footage, responding with 'surprise' when encountering data that conflicts with its understanding.

The innovation, known as Video Joint Embedding Predictive Architecture (V-JEPA) from Meta, starts with no pre-conceived notions about the physics in videos, yet it eventually deciphers how the world operates.

Enhancing Abstraction

In developing AI for self-driving cars, engineers have discovered the difficulty of ensuring systems accurately interpret visual inputs. Systems typically decode videos either by identifying their content or by recognizing object outlines, operating within a 'pixel space' that weighs every pixel equally.

These pixel-focused models are flawed. Consider analyzing a neighborhood street; the system might overemphasize unimportant details like moving leaves, overlooking crucial data like the traffic light's color or a car’s location. 'You want to bypass pixel details since many aren’t essential,' remarked a computer scientist from Brown University.

In 2022, Yann LeCun of NYU and AI research leader at Meta, introduced JEPA for still images. Following this was V-JEPA in 2024, which tackles similar problems in video analysis.

While ordinary systems mask pixels within video frames and train networks to elucidate these hidden areas at pixel levels, V-JEPA instead models high-level abstractions—latent representations—facilitating a more efficient understanding.

Latent representations abstract the core essence of data. Given sketches of cylinders, for example, an encoder can learn to translate these into numerical values indicating properties like height and orientation, simplifying thousands of pixels into key numbers. The decoder then reconstructs the cylinder images from these essential details.

V-JEPA prioritizes forming and reproducing these latent codes. Its architecture includes three main segments: encoder 1, encoder 2, and a predictor. Initially, video frames are provided with masked pixels to encoder 1, generating latent codes. Concurrently, unaltered frames are processed by encoder 2, yielding another set of latent codes.

Here, the predictor functions by utilizing latent codes from masked frames to foresee encoder 2's outcomes. This method allows the system to focus more on meaningful elements—like identifying cars—while disregarding non-essential data such as moving leaves.

Mimicking Intuition

Earlier this year, the V-JEPA creators tested its ability to comprehend intuitive physical characteristics such as object permanence and gravity. In assessments where AI models must judge if video actions are physically plausible, V-JEPA achieved nearly 98% accuracy. In contrast, a pixel-space-based model barely surpassed random guessing.

For autonomous robots, possessing a form of physical intuition is vital for motion planning and interacting with surroundings.

The team evaluated V-JEPA's 'surprise' by measuring discrepancies between expected and actual video frames. Notably, the model highlighted anomalies, such as a ball not reappearing after rolling behind an obstacle, demonstrating an intuitive surprise reminiscent of human infants.

A computational neuroscientist from University College London believes V-JEPA approaches human-like learning, though it lacks comprehensive uncertainty management. For example, when frame information is insufficient, V-JEPA doesn't effectively quantify unpredictability.

In June, Meta unveiled their more advanced V-JEPA version with a model underpinning 1.2 billion parameters, trained on 22 million videos. The model's application in robotics demonstrated how limited new data (around 60 hours from robots) can be used to enhance predictors for robot decision-making.

For advancing V-JEPA, the team devised a more challenging assessment, where the model scarcely exceeded random accuracy. This struggle is comparable to memory limits, highlighted by V-JEPA only handling brief video inputs before forgetting previous details.