World models for dexterous manipulation: prediction is a starting point
Separate predicting a hand–object interaction from choosing an action. Learn what to inspect, measure and compare before using a world model for control.
Separate prediction quality from manipulation success and design two corresponding tests.
The question a world model answers
An action-conditioned world model asks what may happen after an action. Its output could be a future image, hand pose or a learned representation of the scene. A controller or planner must still decide what to do with that prediction. For example, two candidate finger motions might predict different object rotations. Selecting a motion needs a task objective and a way to reject unsuitable predictions.
Different outputs serve different purposes
DexWM studies future latent representations of hand–object interaction; its repository documents training and rollout evaluation. Dexterous World Models (DWM) provides scene- and hand-motion-conditioned video generation. These are useful research references with different representations and evaluation paths. A realistic-looking video is not a force estimate, a collision guarantee or an executable sequence of joint commands. Inspect the output rather than assuming the name “world model” defines a complete control system.
Make contact the test, not the decoration
A hand approaching an object can be easy to predict until contact changes the motion. Test the transitions that matter to your task: first touch, slip, occlusion, regrasp and release. State the prediction horizon and what is observed versus inferred. If tactile signals are unavailable, record that limitation. A model that appears convincing before contact may still give poor predictions during the part of the task that decides success.
Use two separate evaluations
First test prediction on held-out sequences, including errors at multiple horizons and failures around contact. Then ask whether using the model improves completed tasks compared with a policy or planner that does not use it. Keep the available observations and action budget comparable. Improvement in image similarity alone does not establish better manipulation, and task success alone does not explain whether the predictions were physically meaningful.
Avoid evaluation leakage
Neighboring frames of the same interaction share object appearance, lighting and motion. Randomly splitting frames can make a test easier than a new deployment. Choose an episode, session or object split that matches the claim you want to make. Record the source data and its permissions, the software revision, checkpoint and preprocessing. When only simulation is evaluated, label the conclusion as simulation evidence.
A first experiment without hardware
Select a documented public evaluation, inspect one permitted sequence and reproduce its stated output if you have the required assets and compute. Build a small failure ledger: object drift, incorrect hand shape, missed contact, unstable rollout and timing mismatch. Keep examples that fail. Only after the prediction baseline is understood should you propose a separate planning or control experiment with a compatible embodiment.
Three world-model roles, three different outputs
DreamerV3 illustrates latent dynamics used to train an actor-critic policy in imagination. DexWM exposes rollout-prediction evaluation and a separate simulated robot-task evaluation. DWM generates scene/hand-conditioned video. These outputs should not share a single “accuracy” leaderboard. State the representation, prediction horizon and downstream decision, then test the failure mode relevant to that decision.
Keep a living evidence record
A useful research note records the source version, date reviewed, released artifacts, exact experimental setup and unresolved questions. Mark observations as author-reported, reproduced locally, or proposed. Update the note when a checkpoint, license or evaluation changes; keep the older conclusion traceable. The linked profiles are an initial curated set, not an exhaustive census or a hardware compatibility matrix.
What does each result establish?
| Observation | Supports | Does not establish |
|---|---|---|
| Plausible generated video | A visually plausible predicted sequence | Contact forces or task completion |
| Low held-out pose error | Accuracy under the stated data split | Performance on another hand |
| Higher simulated task success | Benefit in the tested simulator | Physical reliability |
| Documented physical trials | Evidence for those tasks and conditions | Universal dexterity |
Work through the mechanics
Explain accumulated prediction error and design a downstream control test.
Prerequisites: Conditional probability, sequential data and the policy/action distinction.
Prediction, belief and control are separate objects
A state transition model predicts a next state given the current state and action. With partial observations, the system instead maintains a history-dependent belief or latent state. An encoder that uses the next real observation during training has information unavailable to an open-loop forecast.
A latent model may have a prior p(zₜ₊₁ | zₜ, aₜ) and an observation-informed posterior q(zₜ₊₁ | history, oₜ₊₁). Reconstruction, reward prediction and regularization can train useful representations, but low training loss does not prove the latent variables capture slip or contact. Dreamer is one example of learning behavior through latent imagination; its full algorithm includes details omitted from this teaching notation.
For a world-model evaluation, state exactly which future observations are available. A teacher-forced one-step prediction resets from real observations at every step. An open-loop rollout feeds predictions forward. These are different tests and should be reported separately.
Policy: aₜ ~ π(a | history) Dynamics: ẑₜ₊₁ ~ p(z | ẑₜ, aₜ) Observation: ôₜ₊₁ ~ p(o | ẑₜ₊₁)
Read the primary work: DreamerV3: learning through latent imagination ↗ · DexWM: prediction and task evaluation ↗
See how a small error can accumulate
Consider a hypothetical recurrence eₖ₊₁ = L eₖ + ε, with e₀ = 0. ε is a fresh error contribution at each step and L describes amplification. If a suitable state-space norm and Lipschitz bound existed, the same expression could describe an upper bound. Here it is only a synthetic recurrence: we have not estimated either quantity for a robot.
At L = 1, error grows linearly as kε. At L < 1, this recurrence approaches ε/(1 − L). At L > 1 it can grow rapidly. Contact transitions can invalidate smooth local assumptions, and an average error can hide rare drops. Use the explorer to separate one-step accuracy from long-horizon reliability; do not interpret its curve as a measured model forecast.
eₖ₊₁ = L eₖ + ε, e₀ = 0 For L = 1: eₖ = kε For L ≠ 1: eₖ = ε(Lᵏ − 1)/(L − 1)
Explore error accumulation
Synthetic teaching example · calculations only. No inference, simulation or hardware execution.
Inspect every rollout step
| Step | Synthetic error |
|---|---|
| 0 | 0.0000 |
| 1 | 0.0200 |
| 2 | 0.0420 |
| 3 | 0.0662 |
| 4 | 0.0928 |
| 5 | 0.1221 |
| 6 | 0.1543 |
| 7 | 0.1897 |
| 8 | 0.2287 |
| 9 | 0.2716 |
| 10 | 0.3187 |
Read the primary work: MIT Underactuated Robotics: system identification ↗
Turn a prediction into a falsifiable decision
A receding-horizon planner can propose several action sequences, roll them out with a model, score predicted outcomes and execute only the beginning of the chosen sequence. Then it observes again. This limits how long the plan runs without new evidence, but does not guarantee stability or avoid model exploitation.
In a synthetic example, candidate A has predicted progress 0.80 and penalty 0.40; B has progress 0.65 and penalty 0.05. With score = progress − 0.5 × penalty, A scores 0.60 and B scores 0.625. B wins despite lower progress. The penalty is a chosen proxy, not a calibrated collision probability. Changing it changes what the planner optimizes.
Compare against the same policy or planner without the learned model. Hold the action budget and task distribution fixed. Report prediction error by horizon and by contact phase, then report real or simulated task success separately. An ensemble’s disagreement is a diagnostic, not a guarantee that all models are wrong in different ways.
while task_active:
observation = read_current_observation()
candidates = propose_action_sequences(observation)
forecasts = model.rollout(observation, candidates)
selected = choose_by_declared_cost(forecasts)
# A real stack also needs feasibility checks and a valid controller.
execute_prefix(selected)
# Reobserve; do not replace feedback with the predicted video.Read the primary work: MIT Underactuated Robotics: trajectory optimization ↗
Audit a world-model result
A paper reproduction has low one-step image error but loses the object during long simulated rollouts. Propose one diagnostic and one downstream comparison. Can a sharp generated video establish manipulation success?
Reveal the worked answer
Plot error at multiple rollout horizons on held-out sequences, separating free motion from contact and releases. Check whether evaluation used real future observations. Compare a fixed-budget controller with and without the model under the same task and reset distribution. Sharp video alone does not establish object-state accuracy, feasible actions or task completion.
Self-review checklist
- Separates teacher-forced and open-loop prediction.
- Measures error by horizon and contact phase.
- Defines a controlled downstream comparison.
- Does not equate visual realism, uncertainty proxies and safety.
Original Dexhands teaching examples. This is a self-study rubric, not automated grading or certification. Research sources reviewed September 25, 2026; examples do not report experiments run by Dexhands.
Try it yourself
EVIDENCE LAB · invented teaching data
The denominator changes the story.
Assisted successes are a subset of all successes here. “Unassisted” alone does not establish autonomy: the control mode and intervention policy still need to be documented. These counts describe a sample, not a guaranteed future success rate.
Your practical task
- Identify whether one release predicts images, poses or latent states.
- Choose a held-out contact transition and a prediction metric.
- Define a separate task-success test and baseline.
What to produce: A prediction evaluation and a separate manipulation evaluation.
Check your understanding
Original sources & next steps
DexWM: official implementation and evaluation ↗Dexterous World Models: official implementation ↗DreamerV3: author implementation ↗Original Dexhands teaching material. Lesson and linked references reviewed 2026-09-25. Research links are not endorsements or evidence of hardware compatibility.