The model
VLSA: vision, language, sensory, action.
An open vision-language backbone gives the robot sight, language and planning. Cerebel adds the sensory half: a touch encoder and an action expert, fused so touch shapes every move.
Architecture
In plain terms: the robot looks, listens to what you ask, and feels what its fingers touch. One model takes all three in and decides how to move.
Most robot AI decides from camera images alone. VLSA adds a sense of touch, and keeps feeling while it moves, so it can correct a slip or confirm a click as it happens instead of finding out afterwards.
Inputs
Head cameras
RGB + depth
Task instruction
Natural language
Touch
48 ch × 5 fingers · 100 Hz
Encoders
Vision-language model
Sight, language, planning · ~3B params
Inherited · open
Sensory encoder
~50M ViT → touch tokens
Ours
Fusion
Shared latent
Cross-modal attention: touch with sight and instruction
Ours
Policy
Action expert
Flow matching, trained on floor data
Ours
Output
Arm + dexterous hand
On-robot, Jetson Thor
Ours
↺ Closed loop at 100 Hz: touch re-enters mid-action
Inherited
A vision-language model of about 3B parameters, for sight, language and planning.
Ours
A touch encoder (about 50M-parameter ViT) that turns tactile data into compact touch tokens, and an action expert trained with flow matching on our floor data.
Train → infer
Trained on NVIDIA RTX 6000 Pro GPUs. Inference runs on the robot (Jetson Thor), closed loop at 100 Hz, with no cloud round trip, so slip is corrected mid-move.
Three model layers
What it learns, how it describes it, how it gets new skills.
Manipulation of Deformable Objects
MODO
What it does
Learns grip, slip, insertion and seating on things that bend.
Status
In development
Object-Centric Action Representation
OCAR
What it does
Describes each manipulation by object motion and finger contact, so human demonstrations and robot demonstrations produce the same description.
Status
In development
Skill acquisition
Auto-Executor
What it does
Turns an instruction into tasks, maps them to skills, and learns missing skills in simulation.
Status
Demonstrated on tote-to-tote picking
OCAR
A human hand and a robot hand, described the same way.
Joint angles differ from hand to hand. What the object does, and where the fingers touch it, does not. Describing a manipulation that way lets one dataset teach many robots.
Human demonstration
One description
- Object motion
- Where the object goes, and how its shape changes
- Finger contact
- Which fingers touch it, where, and how hard
Robot demonstration
Auto-Executor
From an instruction to a finished task.
Auto-Executor breaks an instruction into tasks, maps each task to a skill, and learns any missing skill in simulation. It has been demonstrated on tote-to-tote picking.
Select a scenario to watch it. Everything shown is simulated; no skill in these clips ran on hardware.
01 · Simulated demo
Happy path
An instruction becomes a plan, and each step runs in order: locate the item, pick it, place it.
02 · Simulated demo
Recovery: grasp miss
A grasp misses. The executor notices, retries with a variation, and carries on with the plan.
03 · Simulated demo
Corrective step: slip
An item slips while it is being placed. The executor inserts corrective steps (locate it again, re-pick) and finishes the task.
04 · Simulated demo
Ambiguity
The instruction fits more than one object. The executor resolves which one is meant before it acts.
05 · Simulated demo
Capability gap
Asked to fold a towel, it finds no skill for the job, does nothing unsafe, and logs the missing skill for learning.