Understand or Predict: A Survey of World Models from Recurrent Controllers to the 2026 Frontier

Introduction

A world model predicts what happens next in an environment. Humans rely on one to catch a falling glass. The eyes register the glass, the brain estimates its position and speed, and a prediction of the next half second sends the hand to where the glass will be rather than where it is now. This does not involve solving equations; it is a fast, approximate simulation of the near future used to guide action. A world model does this whether it runs in a brain or in a machine.

The loop has three steps. First, observe: take in the current input, such as pixels from a camera, sound, or the state of a control. Second, represent: compress that raw input into a compact internal state that keeps what matters and discards the rest. Third, predict: from that internal state, given an action or the passage of time, produce an estimate of what comes next. The prediction is then compared against the next observation, and the loop repeats. The representation step is what distinguishes a world model. A camera observes and a screen displays, but neither forms a representation of the environment. A world model builds an internal summary and uses it to predict the near future.

For most of the history of artificial intelligence this idea remained a minor topic in robotics and control. In the last two years it became central. Video generators like Sora, playable generated worlds like DeepMind’s Genie, and self-supervised predictors like Meta’s V-JEPA are all attempts, in different ways, to build a machine that holds a model of its environment (Ding et al., 2024; OpenAI, 2024; Puspitasari et al., 2024). This survey is organized around Ding et al. (2024), Understanding World or Predicting Future?, and extended to cover the 2025 to 2026 frontier that came after it.

The observe-represent-predict loop contains a split that divides the field. A world model can be built to understand its environment, forming a compact internal representation of how it works, or to predict what the environment will look like next, frame by frame, in full detail. These are different objectives and produce different systems. A model built for understanding maintains an abstract summary and does not represent surface detail. It represents that the glass will hit the floor without rendering every reflection during the fall. A model built for prediction generates the next frame at full pixel detail and plans by evaluating its own generated video. The first maintains a compact latent state; the second generates full-detail frames. This survey uses this split as its organizing distinction.

The rest of the survey is structured as follows. The next section makes the understand-versus-predict split precise and defines a shared formal object. The two sections after that cover the understand branch and the predict branch and their main architecture families. A section on applications covers games, robots, driving, and simulated societies. A section on measurement covers how these systems are evaluated and why a single score is misleading. A final section covers open problems and convergence and returns to the title question.

The Two Purposes and a Shared Formal Object

The survey frames the field around one design choice (Ding et al., 2024). One approach builds a world model for understanding: compress the environment into a small internal representation that captures how it works, then use that representation to make decisions. The other builds it for prediction: generate what the environment will look like next, in full detail, and use those generated futures to plan. Both are models of the environment. They differ in what the model is for, and that difference determines the rest of the design.

These are not only engineering preferences; they reflect different positions on where intelligence comes from. In 2018, David Ha and Jürgen Schmidhuber argued that an agent should build a small, abstract internal model and learn to act within its own predictions, prioritizing understanding over pixel detail (Ha and Schmidhuber, 2018a). In 2022, Yann LeCun argued the same position more strongly: a world model must predict future states well enough to plan over them, but should do so in an abstract space rather than in raw pixels, because predicting pixels spends effort on detail that does not matter (LeCun, 2022). Video models like Sora took the opposite approach and predicted the future as high-resolution video (OpenAI, 2024). The understand position holds that useful computation happens in a compressed space and that surface detail is a distraction. The predict position holds that a model able to render the world correctly must have learned how the world works, so fidelity is evidence of understanding.

A system’s purpose is visible in its architecture. The understand branch places an encoder at the front that discards detail, mapping a high-dimensional input into a low-dimensional latent space of a few dozen or few hundred numbers, and runs its dynamics entirely within that latent representation. Nothing is decoded back to pixels for planning. The predict branch makes the output the environment itself, and its claim to understanding is implicit: it must have captured object permanence, approximate physics, and how scenes evolve, or the generated video would be inconsistent. Most systems fall between these poles. A self-driving stack that plans over a top-down occupancy grid is closer to understand. A driving simulator that renders photorealistic video is closer to predict.

Both branches share the same formal object. Decision making with a world model is almost always written as a Markov decision process, the tuple

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, T, R, \gamma),$$

with state space $\mathcal{S}$, action space $\mathcal{A}$, a transition model $T(s' \mid s, a) = \Pr(s_{t+1} = s' \mid s_t = s, a_t = a)$, a reward $R(s,a)$, and a discount factor $\gamma \in [0,1)$. A policy $\pi(a \mid s)$ chooses actions, and the goal is to maximize expected discounted return

$$J(\pi) = \mathbb{E}_{\pi}\!\left[\sum_{t=0}^{\infty} \gamma^{t}\, R(s_t, a_t)\right].$$

A world model is a learned transition model $T$ (and often a learned $R$). The understand-versus-predict split is a disagreement about what the state $s$ should be. The understand branch makes $s$ a small learned latent variable, so $T$ is a low-cost map between compact latent codes. The predict branch effectively makes $s$ the raw observation, so $T$ becomes a full generative model over pixels.

The idea is old. In 1943 the psychologist Kenneth Craik wrote that the mind constructs small-scale models of reality and runs them to anticipate events before they happen (Craik, 1943). Four decades later Philip Johnson-Laird developed this into a theory of mental models: internal representations that preserve the structure of a situation, so that inspecting the model yields true statements about the world (Johnson-Laird, 1983). Marvin Minsky’s 1974 frames gave knowledge an explicit structure, templates for stereotyped situations with fillable slots (Minsky, 1974). The differentiable version arrived in 1990, when Schmidhuber proposed training one recurrent network to predict the world and a second to act within those predictions, so the acting network could improve without interacting with the real environment (Schmidhuber, 1990). Almost nothing in modern world models is conceptually new. What changed is that these models can now be learned from data at scale instead of built by hand.

The Understand Branch: Latent-State World Models

Model-based planning

World models address one problem: how an agent can plan by simulating consequences instead of learning only by trial and error in the real environment. An agent that learns actions directly, mapping each situation to an action tuned by reward, is model-free. An agent that instead learns how the environment works, producing a learned simulator it can run forward, is model-based. With that simulator it can plan: evaluate candidate actions in simulation, select the one whose predicted outcome is best, and then act in the real environment. Simulation is cheap and carries no real-world risk. The limitation is that simulated outcomes are only as accurate as the learned model, and most of the engineering in this field is about making that model accurate enough to plan against.

A model-based agent fits a parametric model $M_\theta(s' \mid s, a) \approx P(s' \mid s, a)$ and plans against it in place of the true dynamics $P$, which it does not observe (Ding et al., 2024). The simplest fitting method regresses the next state with mean-squared error, and its behavior has consequences for the whole field. Fix a state and action and let $s'$ be the random next state. A deterministic predictor outputs a single point $\hat{s}$, and its expected squared error is minimized at

$$\hat{s}^\star = \arg\min_{\hat{s}} \, \mathbb{E}\big[\lVert s' - \hat{s}\rVert_2^2\big] = \mathbb{E}[s'].$$

The optimal single prediction is the mean of the possible futures. When those futures are multimodal and far apart, that mean is a low-probability point, a state that may never occur. In pixels, the mean of several sharp frames is a blur. This is the basis of the pixel-versus-latent split that recurs through the survey. The alternative is to predict a distribution, fitting it by minimizing $\mathrm{KL}(P \Vert M_\theta)$, so the model represents its uncertainty rather than collapsing the outcomes to one value.

Given a learned model there are two standard ways to derive behavior. Model predictive control rolls several candidate action sequences forward a short horizon, scores each by predicted return, executes only the first action of the best sequence, then replans from the new real state. Replanning every step keeps small model errors from compounding over a long predicted trajectory. Tree search, used in AlphaGo and MuZero, instead grows a branching tree of predicted continuations and allocates computation to the promising branches (Schrittwieser et al., 2020). MuZero’s learned model never predicts the next observation. It predicts only the quantities planning needs, a reward, a value, and a next hidden state, and that hidden state is a learned latent that never decodes back to pixels. This is the value-equivalent approach: the model is justified by producing correct decisions rather than correct images. It is the same distinction that separates the latent planners here from the full-detail video models discussed later.

The recurrent world model

In 2018 Ha and Schmidhuber’s World Models implemented this loop in a small system and gave the field its name (Ha and Schmidhuber, 2018a; Ha and Schmidhuber, 2018b). It divides the problem of acting in an environment observed only through pixels into three components: Vision, Memory, and Controller. Vision is a convolutional variational autoencoder that compresses each frame into a low-dimensional latent, a 32-number vector in the paper, trained only to reconstruct frames from a random policy. Memory is a recurrent network with a mixture-density output, an MDN-RNN, that takes the current latent and action and predicts a distribution over the next latent. The Controller is a single linear layer that maps the current latent and Memory’s hidden state to an action.

The VAE maximizes the evidence lower bound, a reconstruction term minus a penalty that keeps the latent close to a simple prior:

$$\mathcal{L}_{\text{VAE}} = \mathbb{E}_{q_\phi(z \mid x)}\!\left[\log p_\theta(x \mid z)\right] - D_{\mathrm{KL}}\!\big(q_\phi(z \mid x)\,\|\,p(z)\big), \qquad p(z) = \mathcal{N}(0, I).$$

Memory models the next latent as a mixture of $K$ Gaussians, $P(z_{t+1} \mid a_t, z_t, h_t) = \sum_k \pi_k \mathcal{N}(z_{t+1}; \mu_k, \sigma_k^2 I)$, trained by maximum likelihood. A single Gaussian would force one future, and averaging a left turn and a right turn produces a point between them that never occurs. A mixture keeps the modes separate. This explicit representation of uncertainty is what makes Memory usable for generating rollouts.

This is the technique that named the field. Because Memory can generate the next latent on its own, Vision can be disconnected from the real camera and Memory can feed its own predictions back as input. The agent then runs forward entirely within Memory’s predictions, and the Controller is trained on those predicted rollouts. The original demonstration trained its entire policy inside a generated car-racing game and a VizDoom shooter, then transferred it to the real games. A temperature parameter rescales the sampled variance, $\hat{z}_{t+1} \sim \sum_k \pi_k \mathcal{N}(\mu_k, \tau \sigma_k^2 I)$. Increasing it makes the generated rollouts more variable and harder to act in, which forces the Controller to learn a policy that tolerates noise rather than one that only works in a deterministic rollout. Because the Controller is small, a few hundred parameters, it is trained without gradients, using the CMA-ES evolution strategy, which handles sparse, noisy reward.

Dreamer and latent imagination

The Dreamer line scaled the approach of training a policy inside a learned model until a single agent with one fixed set of hyperparameters solved more than 150 tasks and obtained a diamond in Minecraft from scratch, a result published in Nature in 2025 (Hafner et al., 2023). Its core component is the recurrent state-space model, or RSSM, introduced in PlaNet (Hafner et al., 2019). It splits the latent state into two parts. The deterministic part $h_t$ is a noise-free running summary of the past, produced by a GRU. The stochastic part $z_t$ is a small random vector that captures what is genuinely uncertain at the current step, the component that cannot be predicted from memory and must be observed.

The RSSM maintains two estimates of the stochastic part. The prior $p_\theta(z_t \mid h_t)$ predicts the next latent from memory alone. The posterior $q_\theta(z_t \mid h_t, o_t)$ is the corrected estimate after the observation arrives. Training drives the prior toward the posterior, which is equivalent to training the model to predict the next observation. The world model is fit by maximizing a per-timestep lower bound,

$$\mathcal{L}_t = \mathbb{E}_{q_\theta}\!\big[\log p_\theta(o_t \mid s_t) + \log p_\theta(r_t \mid s_t) + \log p_\theta(c_t \mid s_t)\big] - \beta\, \mathrm{KL}\!\big[\, q_\theta(z_t \mid h_t, o_t)\,\big\|\,p_\theta(z_t \mid h_t)\,\big],$$

where three small heads decode the observation, the reward, and a continue flag from the state $s_t = (h_t, z_t)$. Once the prior is accurate, the agent can run the model forward on priors alone, with no observations. A rollout generated this way factorizes entirely into prior terms:

$$p_\theta(z_{t+1:t+H} \mid s_t, a_{t:t+H-1}) = \prod_{k=1}^{H} p_\theta(z_{t+k} \mid h_{t+k}), \qquad h_{t+k} = f_\theta(h_{t+k-1}, z_{t+k-1}, a_{t+k-1}).$$

The posterior never appears, because there is no observation to condition on. The actor and critic are trained only on these latent rollouts. Real frames are used only to fit the world model, never to train behavior directly. This is why latent rollouts are cheap: a rollout is a sequence of small recurrent steps with no image generation.

The difficulty was not generating one future but building a single agent that works across very different problems. A balancing pole gives small fractional rewards every step, while Minecraft gives almost no reward for hours and then a large one. DreamerV2 moved to categorical latents with a straight-through estimator and added KL balancing (Hafner et al., 2021). DreamerV3 added the components that made one configuration work across all tasks: the symlog transform $\operatorname{symlog}(x) = \operatorname{sign}(x)\ln(1 + |x|)$, which compresses large magnitudes while staying linear near zero; percentile-based return normalization; and a free-bits floor on the KL that prevents the model from over-compressing the latent and discarding rare signals (Hafner et al., 2023). These are why one hyperparameter set covers tasks from pole-balancing to diamond-mining. Dreamer does not render the future at full detail. It optimizes behavior in a compact latent space and decodes pixels only for validation during training.

JEPA: predict the representation, not the pixels

LeCun argues that predicting the future in pixels is the wrong problem, for the reason above. When many futures are possible, a model trained to match every pixel produces their average, which resembles none of them. This is not a flaw that more data fixes; it is the direct result of minimizing pixel error. LeCun’s alternative is to change the target. Instead of generating the next frame, the model predicts a compact description of it, produced by a learned encoder that discards whatever does not matter. Predicting only this compact description leaves far less to be uncertain about, and therefore far less to average away.

This is the joint-embedding predictive architecture, or JEPA (LeCun, 2022). It does not generate. An encoder $f_\theta$ maps the context to a representation, a target encoder $f_{\bar\theta}$ maps the part to be predicted to its own representation, and a predictor $g_\phi$ maps the first to an estimate of the second. The loss is computed entirely in representation space:

$$\mathcal{L}_{\text{pred}} = \big\lVert\, g_\phi(f_\theta(x), z) - \operatorname{sg}(f_{\bar\theta}(y)) \,\big\rVert_1,$$

where $z$ is a conditioning variable indicating which target is being predicted and $\operatorname{sg}$ is a stop-gradient. The main failure mode is representation collapse: both encoders output the same constant vector for every input, making the loss zero. Contrastive learning avoids this by including many negative pairs and pushing them apart, but that requires a large, carefully selected set of negatives. JEPA uses no negatives. The target encoder receives no gradient and tracks the online encoder by an exponential moving average, $\bar\theta \leftarrow \tau \bar\theta + (1 - \tau)\theta$. Because the target is not optimized and changes slowly, the constant solution is no longer a trivial minimum of the loss. Where the slow target is insufficient, VICReg-style variance and covariance terms constrain the batch of representations to have spread and to be decorrelated, which penalizes collapse without requiring negatives (Bardes et al., 2024).

I-JEPA applied the scheme to images by predicting the representations of masked-out regions (Assran et al., 2023). V-JEPA extended it to video, masking regions of space and time (Bardes et al., 2024). V-JEPA 2 scaled this to more than a million hours of internet video, and the frozen video encoder can drive a real robot arm zero-shot (Assran et al., 2025). Control is formulated as energy minimization over action sequences,

$$a_{1:H}^{\star} = \arg\min_{a_{1:H}} \; \big\lVert\, P(s_x, a_{1:H}) - z_{\text{goal}} \,\big\rVert^2,$$

solved with the cross-entropy method and executed one action at a time, which is standard model predictive control run in representation space. Given a goal image, the system generates candidate short action sequences in latent space, scores each by how close its predicted endpoint is to the goal representation, and executes the best one, with no robot-specific reward and no fine-tuning on the task. It is the strongest evidence so far that a robot can plan against a learned model without generating any images of the future.

World knowledge inside language models

Another line in the understand branch obtains a model of the world as a byproduct of language training. A large model trained only on text never receives a map, a clock, or a falling object as input, yet a substantial amount of the world’s structure is encoded in it (Ding et al., 2024). The survey sorts this into knowledge of the global physical world (geography and time), the local physical world (how a room is laid out), and human society (other people and their beliefs). A text-trained model answers which of two cities is further north correctly far more often than chance, and it can order the printing press before the telephone.

The evidence is structural rather than only behavioral. When a language model processes text, every token becomes an activation vector in a space of thousands of dimensions, and specific real-world quantities appear as directions in that space. Fitting a linear probe $\hat{y} = w^\top h(x) + b$ by least squares from activations to a location’s latitude, and recovering held-out coordinates with high $R^2$, indicates that latitude is encoded as an approximately linear direction. This has been found for latitude, longitude, and time. The model encodes an internal coordinate frame it was not trained to produce. Two caveats bound the claim. A high $R^2$ shows the coordinate is linearly decodable, but decodability is not a dynamics model. Nothing here defines a transition $T(s' \mid s, a)$. The probe shows that the model stores a coordinate, not that it can predict the environment forward, which is the distinction between this background knowledge and simulator-style world models. The two are complementary: a language-driven agent trained in its environment can plan against a learned dynamics model, and when it has not been trained there, this background knowledge is the fallback. The knowledge is present but unreliable in the details, so it should be treated as a prior rather than an accurate model.

The Predict Branch: Generative World Models

Video as a simulator

The predict branch takes the question literally. If a world model is meant to predict the future, it predicts every pixel of every frame directly. OpenAI’s Sora technical report states this in its title, Video generation models as world simulators (OpenAI, 2024). The claim is that a video model trained on enough footage, at large enough scale, moves from reproducing clips to capturing how the physical world behaves. There are two reasons this is worth the cost of predicting pixels. Video is available in nearly unlimited supply and requires no labeling. And pixels require explicit detail: a compact latent representation can omit whatever it treats as unimportant, while a video model must specify where every object is and how the light falls.

The underlying method is the diffusion model, built from two paired processes (Ho et al., 2020). The forward process is fixed and requires no learning: starting from a clean image, add a small scheduled amount of noise repeatedly until the result is indistinguishable from noise. Because every step is Gaussian, the chain has a closed form that maps directly from the clean sample to any step,

$$x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1 - \bar\alpha_t}\,\bar\epsilon, \qquad \bar\epsilon \sim \mathcal{N}(0, \mathbf{I}), \qquad \bar\alpha_t = \prod_{s=1}^{t}(1 - \beta_s),$$

which makes training cheap because a noisy sample at any step is one draw. The network learns the reverse process by predicting the noise, with the plain objective

$$\mathcal{L} = \mathbb{E}_{t, x_0, \epsilon}\Big[\big\lVert \epsilon - \epsilon_\theta(x_t, t)\big\rVert^2\Big].$$

Running this loop directly on full-resolution pixels is very slow, so latent diffusion moves the process into a smaller space: a VAE compresses images into a compact latent grid, diffusion denoises the latents, and only the final latent is decoded (Rombach et al., 2022). A condition, a text prompt or an action, is injected with classifier-free guidance, which trains the model to denoise both with and without the condition and then extrapolates at sampling time (Ho and Salimans, 2022):

$$\tilde{\epsilon} = (1 + w)\,\epsilon_\theta(x_t, t, c) - w\,\epsilon_\theta(x_t, t, \varnothing).$$

For video the latent grid becomes a stack of grids, one per frame, and the model denoises the whole stack at once so the frames are consistent. Sora divides the video latent into spacetime patches and applies a diffusion transformer that attends across both space and time, which produces long-horizon consistency: an object that leaves the frame and returns, lighting that holds steady over many seconds (OpenAI, 2024). None of this puts physics in the objective. The model is trained to produce frames that match the training data, not frames that conserve momentum, which is the gap the physics section addresses.

Interactive and autoregressive worlds

A generated video functions as a world rather than a recording when it can be controlled. DeepMind’s Genie generates controllable 2D worlds from unlabeled internet video (Bruce et al., 2024). Given a single starting image, it can be played, and Genie inferred the effect of the controls without any action labels. The training videos had no button or joystick logs, only raw footage. Genie has three parts. A spatiotemporal tokenizer compresses raw frames into a grid of discrete tokens. A latent action model reads consecutive frames and emits a single action code from a small codebook, on the order of eight possible actions. A dynamics model predicts the next frame’s tokens given the past and that inferred action.

The design depends on one bottleneck. The latent action model is trained in a reconstruction loop where the only path from the future into the prediction runs through the action code $\tilde a_t \in \{1, \dots, A\}$ with $A$ small. A discrete channel of $\log_2 A = 3$ bits cannot transmit the full next frame. It can transmit only which of a small number of transition modes occurred, and the loss favors the modes that recur across the data and explain the most variance, such as moving in each direction and jumping. The bottleneck therefore does more than compress the action; it selects for the small set of repeatable, controllable changes. At play time the latent action model is discarded and a human supplies one of those codes directly, which is why the learned actions function as discrete controls. The dynamics model is trained with a cross-entropy objective over the token vocabulary, $\mathcal{L}_{\text{dyn}} = -\sum_t \log P_\theta(z_t \mid z_{<t}, \tilde a_{<t})$, and generates frames with MaskGIT-style iterative decoding that fills in tokens over a few parallel rounds rather than strictly left to right, which is fast enough for interaction.

This is the second major architecture for video world models, and it makes the opposite trade-off from diffusion. A diffusion model constructs an entire frame from noise over many steps, which is expensive per frame but produces high visual quality. An autoregressive model like Genie generates the next frame conditioned on the past and the chosen action, and conditioning on an action matches the structure of a game loop. Genie 2 extended the approach to 3D, generating navigable worlds with consistent geometry from a single image (Google DeepMind, 2024). Genie 3 reached real-time interaction at 720p, sustaining minutes-long navigable worlds in live play, the first general-purpose world model at that scale, presented as a step toward training embodied agents inside a large supply of generated environments (Parker-Holder et al., 2025). A related system, UniSim, learned an interactive real-world simulator that conditions generation on actions across a wide mix of data (Yang et al., 2024).

Embodied 3D worlds

A video is not a navigable space. It follows one path through a scene, and the viewer cannot take a different path or return to check whether a previously seen object is still present. A world model that an agent operates within must support movement through it. The difficulty is not making any single frame look good, which video generators already do, but keeping the scene consistent as the viewpoint changes. When the view pans away from a table and back, it must be the same table, and a model that allows the path to be chosen must retain the layout of the scene, or objects will move between views.

A consistent space requires an explicit geometry rather than pixels alone. Consistency here means that the rendered view depends only on the current viewpoint, not on the path taken to reach it. A frame-by-frame generator renders the view as a function of the entire trajectory $\tau$, so two different paths ending at the same pose $v$ can produce different views, $R_{\text{video}}(\tau) \neq R_{\text{video}}(\tau')$, which is the objects moving between views. Storing an explicit geometry $G$ and rendering with a function of only the stored geometry and the current pose removes the dependence on history, $R(v) = \operatorname{render}(G, v)$, making path-independence hold by construction. Two representations are common: a mesh stores surfaces as connected triangles, and Gaussian splatting stores the scene as millions of small colored semi-transparent blobs, which are fast to render and represent soft detail such as foliage well. Both store the geometry once and render it many times.

Three systems define this frontier. World Labs’ Marble converts a single prompt or photo into a persistent, downloadable 3D world that can be navigated and edited, described as spatial intelligence, using a generative stage that produces candidate views followed by a reconstruction stage that solves for the splats or mesh (World Labs, 2025). NVIDIA Cosmos provides open-weight world foundation models aimed at robots and self-driving cars, with an emphasis on physical behavior, contact, collision, and mass, so that a policy trained against them does not learn incorrect dynamics (NVIDIA, 2025). This matters beyond demonstrations because an embodied agent needs an environment in which to train. A robot cannot be allowed to fail ten thousand times in a real kitchen, and ten thousand real near-crashes cannot be staged to train a car. A generated environment that stays consistent, obeys the physical rules, and can be created cheaply serves as a training environment, and the more consistent it is, the more of what the agent learns there transfers to the real world.

The physics question

This is a central critique in the field: a video can be photorealistic and physically impossible at the same time. A model can render a glass of water with correct light and reflections and still show it pouring so that the liquid hangs in the air. Appearance and physical correctness are evaluated by different standards. Human vision is a lenient standard. A physics engine checks whether mass is conserved, whether momentum is conserved, and whether a dropped ball falls at the rate gravity specifies. A video model trained only to produce plausible frames is scored against the lenient standard of human vision, so it becomes accurate on appearance while remaining weak on physics.

There is a stronger version of this result. When these generators are tested on situations close to their training data the videos are accurate, and when they are tested on new combinations absent from the training set, the physics is wrong (Kang et al., 2024). The controlled experiment builds simple scenarios with known governing equations, trains a generator on one region of the parameter space, and tests in-distribution, combinatorial (seen elements, unseen combinations), and fully out-of-distribution. In-distribution error is low and decreases with scale. Combinatorial and out-of-distribution error remain high. In a failed case, the predicted trajectory follows the nearest training example along whichever property is easiest to copy, color, size, or velocity, rather than the property the true dynamics require. This is the signature of case-by-case retrieval: the model stores a lookup table of observed motions rather than inducing the governing rule. Scaling improves only appearance matching, because the training loss references frame resemblance, not mass, momentum, or energy, so gradient descent allocates capacity to memorizing and interpolating the appearance of observed motions.

Two structural facts explain this. The multimodal-future argument from the planning section applies directly: a squared-error objective on the next frame collapses a multimodal future to its mean $\hat{x}^\star = \mathbb{E}[x]$, which as a trajectory obeys no single law because it is the average of several distinct lawful trajectories. And rollout error compounds. With one-step error bounded by $\epsilon$ and the learned model $L$-Lipschitz, the accumulated error after $H$ steps satisfies $e_{t+1} \le \epsilon + L\,e_t$, which unrolls to

$$e_H \le \epsilon\,\frac{L^{H} - 1}{L - 1}.$$

In the marginal regime $L \approx 1$ this reduces to $e_H \lesssim \epsilon H$, linear in the horizon, and any $L > 1$ makes the error grow exponentially. A locally accurate model can still diverge far from physical reality over a long generated clip, which is why physics violations are most visible in long-horizon prediction. The proposed fix keeps generative models but does not require them to learn physics entirely from data. One approach assigns the physical reasoning to a physics engine and uses the generative model to render the result in realistic pixels. Another adds physical constraints to training, penalizing a clip that violates conservation the same way a blurry clip is already penalized (Kang et al., 2024).

World Models in the Wild

The two branches appear across four application domains, each testing a different capability. Games are the most controlled setting for this comparison: the rules are fixed, the inputs are discrete button presses, and the game engine provides the true next frame, so any prediction can be checked against ground truth. GameNGen ran the shooter DOOM at around twenty frames a second with no game engine, generating every frame with a diffusion model conditioned on the recent gameplay and the current button press, and human raters could barely distinguish it from the real game (Valevski et al., 2024). Its central engineering problem is error accumulation, the main failure mode of any system that generates one step at a time. GameNGen addresses it by corrupting the conditioning frames with noise during training so the model becomes a contraction on its own errors, keeping the effective local sensitivity below one so per-step errors are attenuated rather than amplified. DIAMOND showed that agents can be trained by reinforcement learning entirely inside a diffusion world model and still perform in the real game (Alonso et al., 2024).

Robots cannot learn the way a game agent does. A failed grasp drops a part, a bad step can damage a joint, and the real world runs in real time, so a million attempts cannot be run quickly. A world model moves most of the trial-and-error into simulation. It uses a world model for three tasks: generating synthetic trajectories to expand a small set of expensive demonstrations into many, planning by simulated rollout, and reducing the sim-to-real gap by learning the dynamics from real interaction rather than relying on a hand-built simulator matching reality (Ding et al., 2024). DayDreamer ran a DreamerV3-style agent directly on physical robots and had a quadruped learn to walk from scratch in about an hour of real experience, with no simulator in the loop (Hafner et al., 2023). Planning by simulated rollout has a compact form: encode observations with $\phi$, roll a learned latent model $P$ forward under a candidate action sequence, and choose the sequence whose predicted endpoint is nearest the encoded goal, $a_{1:H}^\star = \arg\min_{a_{1:H}} \lVert \hat s_H - g \rVert^2$. Everything is computed in latent space, which is why the quality of the encoder determines performance, and V-JEPA 2 runs exactly this loop to drive a real arm zero-shot (Assran et al., 2025).

Cities are the domain of self-driving cars. A self-driving stack perceives, predicts, and acts ten or twenty times a second, and the prediction step has become a generative model of the future. Rather than predict raw pixels, modern systems compress the scene into a structured latent, a bird’s-eye-view map or a 3D occupancy grid, and forecast that representation forward as a learned transition model, which the planner searches over with model predictive control. The main advantage is handling rare events. Enough real crashes cannot be collected to train against, but a controllable model conditioned on a scenario description can sample the hazardous cases directly, $s \sim p_\theta(s \mid c)$ for $c$ a rare hazardous condition, replacing the very low probability of a real crash with an event that can be generated on demand (Ding et al., 2024). The caveat is that the synthesized state inherits whatever physics the model learned, with no guarantee it is correct in the tail. The driving world models have since merged with the world-foundation-model work, with Cosmos providing the controllable-video method as a general simulator aimed at generating the scenarios real fleets cannot collect (NVIDIA, 2025).

Societies are also modeled as worlds. The survey’s most abstract domain omits physics and models a community of people, populating a simulated town with characters each run by a language model, giving each a memory and a goal, and having them interact (Ding et al., 2024). A simulated society performs two modeling tasks at once. The macro task treats the whole simulation as an explicit world model of a society, whose emergent properties, an information cascade, a social network forming, a price settling out of many decisions, result from the interactions rather than being programmed. The micro task is that each agent runs its own implicit model of the other agents, a theory of mind. The standard generative-agent architecture combines a memory stream, a retrieval step that scores memories by recency, importance, and relevance, and a reflection-and-planning step that synthesizes higher-level conclusions back into memory. The main difficulty is evaluation. A society has no single next frame to score against, and language-model agents tend toward agreeableness, so a simulated forum can converge to a consensus no real community would reach. The same emergence that makes these simulations useful also makes them hard to validate.

Evaluation and Measurement

If a world model can look correct and be physically wrong, then visual quality alone is not a sufficient measure. Realism, consistency, and physics have to be scored separately, because they are not correlated (Ding et al., 2024). A world model can be scored on five separate dimensions whose scores rarely move together. Visual quality measures whether each frame looks real, where current models are strong. Temporal consistency measures whether the scene stays coherent as it runs; short clips hold together and long ones drift. Physics measures whether the world obeys physical law, the weak dimension that the newer benchmarks were built to expose. Controllability measures whether the world responds correctly to a commanded action. Diversity measures whether the model can produce genuinely different plausible futures or collapses to the average of its training set. The spread between these scores is the important quantity, and a single averaged number hides the gap that determines whether a video model functions as a world model.

The benchmarks sort into three families. Video-centric simulation quality asks whether generated footage matches the statistics of real footage. The classic instrument is Fréchet Video Distance, which fits a Gaussian to real and generated features in a pretrained feature space and measures the distance between them,

$$\mathrm{FVD} = \lVert \mu_r - \mu_g \rVert_2^2 + \operatorname{Tr}\!\Big( \Sigma_r + \Sigma_g - 2\big( \Sigma_r \Sigma_g \big)^{1/2} \Big).$$

The structural weakness is explicit: FVD is a function of second-order statistics of an appearance feature space only, so a generator can drive it toward zero while still violating conservation laws frame to frame. The second family, physical and spatial reasoning, scores what a clip does rather than how it looks. Physics-IQ runs a conditional-prediction protocol: condition the model on the real prefix, have it generate the continuation, and compare against the true future on where and when things moved rather than on appearance (Motamed et al., 2025). It scores motion with an intersection-over-union over masks of moving pixels, $\mathrm{IoU}_t = |M_t \cap \hat{M}_t| / |M_t \cup \hat{M}_t|$, and keeps a pixel-error term separate so a model cannot obtain a good physics score with sharp but misplaced motion. VBench-2.0 reports no single number, scoring a clip on a vector of intrinsic-faithfulness dimensions, commonsense, human motion, physical correctness, each by its own evaluator, so a model with high appearance and low physics appears as an uneven vector that a scalar would have averaged away (Zheng et al., 2025). The third family, embodied decision-making, evaluates the model not on pixels but on whether an agent that plans inside it acts well in the real environment, which is the most demanding test.

The empirical result across these benchmarks is consistent: the strongest open and commercial video generators score high on visual quality, score partially on temporal consistency, and remain weak on physics, with the physics score only loosely correlated to the realism score (Motamed et al., 2025; Zheng et al., 2025). The dissociation follows from what the training gradients optimize. A video generator is trained to reduce the difference between generated and real frames in some feature space, which concentrates capacity on appearance because appearance is what the loss measures, while nothing in a standard FVD-style target references mass, momentum, or contact forces. The model can therefore raise visual quality while physical correctness stays low.

Open Problems and the Convergence

The survey ends with the open problems, which double as a research agenda, and they split into two kinds (Ding et al., 2024). Some are capability gaps that harder data or better methods will eventually close. Others are about responsibility rather than capability, and no amount of scaling solves them.

Four capability problems recur. Reasoning about situations the model has never seen, especially rare and dangerous ones, is the out-of-distribution problem, and its safety-critical version is counterfactual prediction. The structural difficulty is a mismatch between where the data is dense and where the stakes are high: training samples cluster on routine cases, while the collisions and near-misses that determine whether a system is deployable are scarce. Believable human behavior is hard to generate and harder to evaluate, because there is no physics engine for whether a person acted as a person would. The sim-to-real gap is the difference between a model’s behavior inside its generated environment and its behavior once that environment is the real world, addressed by widening the training distribution and grounding the generated world in real physics. And efficiency is inherent to autoregressive generation: because frame two depends on frame one, the computation is sequential and cannot be parallelized across hardware, and a real-time world needs a new frame every few tens of milliseconds indefinitely.

The responsibility concerns attach to specific parts of the pipeline. Data privacy is a property of the training set, since world foundation models train on web-scale data not collected with consent, which the model can memorize and reproduce. Low-cost rehearsal of harm is a property of generality: the same model that can simulate any plausible scenario can simulate a harmful one, and there is no clean structural fix because the capability that enables the misuse is the same one that makes the model useful. Accountability for synthetic media is a property of the output; the deepfake problem now extends to motion and is moving toward three dimensions. Watermarking and provenance tracking are partial measures, useful but not complete, because a watermark can be removed by re-encoding and provenance only helps for media that carries it. The capability frontier and the responsibility frontier are advancing together, and the survey’s position that the second is not subordinate to the first has held up.

Discussion

Return to the first question: understand the world, or predict the future? Based on where the work is in 2026, this was a false choice. A model that only understands produces no observable output: it forms an internal account of the future that it cannot display, verify against human perception, or allow interaction with. A model that only predicts generates a plausible future without a compact theory of why that future follows rather than another, so it diverges, produces inconsistent output, and violates physical law as soon as it is pushed past its training data. Neither one alone is the goal.

The strongest systems combine the two purposes, converging from both directions. From the understand side, the addition is prediction and action. V-JEPA 2 is a model trained in the abstract, pixel-discarding way that nonetheless predicts how the environment will change and uses that prediction to plan a real robot’s actions, an understand-branch model performing a predict-branch task in its latent representation (Assran et al., 2025). From the predict side, the addition is a compact, physically consistent model underneath the pixels. The generative world models built for robots and self-driving cars are no longer required only to look convincing but are anchored to real physics and real sensor data, so that rendering the scene correctly requires modeling the dynamics correctly, and the benchmarks built to evaluate them score gravity, collision, and object permanence rather than only sharpness (NVIDIA, 2025; Kang et al., 2024; Motamed et al., 2025). When this anchoring works, prediction becomes a form of understanding.

The two approaches reach the same target from opposite directions. The understand branch is adding prediction and action, and the predict branch is adding physical consistency and grounding. Both are converging on a single system: one with a compact internal model of how the world works and the ability to run it forward into a viewable, controllable, physically consistent future. Understand and predict are not two kinds of world model but two components of one, which is why these models are currently central to the question of what intelligence is and how to build it.

References

Alonso, E., et al. (2024). Diffusion for World Modeling: Visual Details Matter in Atari (DIAMOND). NeurIPS 2024. arXiv:2405.12399.

Assran, M., et al. (2023). Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (I-JEPA). CVPR 2023. arXiv:2301.08243.

Assran, M., et al. (2025). V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning. Meta AI. arXiv:2506.09985.

Bardes, A., et al. (2024). Revisiting Feature Prediction for Learning Visual Representations from Video (V-JEPA). Meta AI. arXiv:2404.08471.

Bruce, J., et al. (2024). Genie: Generative Interactive Environments. ICML 2024 (DeepMind). arXiv:2402.15391.

Craik, K. (1943). The Nature of Explanation. Cambridge University Press.

Ding, J., Zhang, Y., et al. (2024). Understanding World or Predicting Future? A Comprehensive Survey of World Models. ACM Computing Surveys (2025). arXiv:2411.14499. doi:10.1145/3746449.

Google DeepMind (2024). Genie 2: A Large-Scale Foundation World Model. https://deepmind.google/research/publications/60474/.

Ha, D., and Schmidhuber, J. (2018a). World Models. arXiv:1803.10122 (interactive companion at worldmodels.github.io).

Ha, D., and Schmidhuber, J. (2018b). Recurrent World Models Facilitate Policy Evolution. NeurIPS 2018.

Hafner, D., et al. (2019). Learning Latent Dynamics for Planning from Pixels (PlaNet). ICML 2019. arXiv:1811.04551.

Hafner, D., Lillicrap, T., Ba, J., and Norouzi, M. (2020). Dream to Control: Learning Behaviors by Latent Imagination (DreamerV1). ICLR 2020. arXiv:1912.01603.

Hafner, D., et al. (2021). Mastering Atari with Discrete World Models (DreamerV2). ICLR 2021. arXiv:2010.02193.

Hafner, D., Pasukonis, J., Ba, J., and Lillicrap, T. (2023). Mastering Diverse Domains through World Models (DreamerV3). arXiv:2301.04104; published in Nature 640, 647-653 (2025). doi:10.1038/s41586-025-08744-2.

Ho, J., Jain, A., and Abbeel, P. (2020). Denoising Diffusion Probabilistic Models. NeurIPS 2020. arXiv:2006.11239.

Ho, J., and Salimans, T. (2022). Classifier-Free Diffusion Guidance. NeurIPS 2021 Workshop. arXiv:2207.12598.

Johnson-Laird, P. N. (1983). Mental Models: Towards a Cognitive Science of Language, Inference and Consciousness. Cambridge University Press.

Kang, B., et al. (2024). How Far is Video Generation from World Model: A Physical Law Perspective. ICML 2025. arXiv:2411.02385.

LeCun, Y. (2022). A Path Towards Autonomous Machine Intelligence. OpenReview preprint v0.9.2.

Minsky, M. (1974). A Framework for Representing Knowledge. MIT AI Laboratory Memo 306.

Motamed, S., et al. (2025). Do Generative Video Models Understand Physical Principles? (Physics-IQ). WACV 2026. arXiv:2501.09038.

NVIDIA (2025). Cosmos World Foundation Model Platform for Physical AI. arXiv:2501.03575.

OpenAI (2024). Video Generation Models as World Simulators (Sora technical report). https://openai.com/index/video-generation-models-as-world-simulators/.

Parker-Holder, J., Fruchter, S., et al. (2025). Genie 3: A New Frontier for World Models. DeepMind. https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/.

Puspitasari, F. D., et al. (2024). Sora as a World Model? A Complete Survey on Text-to-Video Generation. arXiv:2403.05131.

Rombach, R., et al. (2022). High-Resolution Image Synthesis with Latent Diffusion Models. CVPR 2022. arXiv:2112.10752.

Schmidhuber, J. (1990). Making the World Differentiable: On Using Fully Recurrent Self-Supervised Neural Networks for Dynamic Reinforcement Learning and Planning. Technical Report FKI-126-90, TU München.

Schrittwieser, J., et al. (2020). Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model (MuZero). Nature 588, 604-609. arXiv:1911.08265.

Valevski, D., Leviathan, Y., Arar, M., and Fruchter, S. (2024). Diffusion Models Are Real-Time Game Engines (GameNGen). Google. arXiv:2408.14837.

World Labs (Li, F.-F., et al.) (2025). Marble: Generating Persistent 3D Worlds (Spatial Intelligence). https://www.worldlabs.ai/.

Yang, S., et al. (2024). Learning Interactive Real-World Simulators (UniSim). ICLR 2024 (Oral). arXiv:2310.06114.

Zheng, D., et al. (2025). VBench-2.0: Advancing Video Generation Benchmark Suite for Intrinsic Faithfulness. arXiv:2503.21755.