BlogTechnology4 min read
Why vision fails at the point of contact
Five everyday tasks with objects that bend, and what a camera misses in each. A plain walk through where touch carries the information vision can't.
Cameras are very good at telling a robot what is in front of it. They are much worse at telling it what is happening under its fingers. For rigid objects, that gap is often small enough to ignore. For objects that bend, it decides whether the task works. This post walks through five tasks we think about every day, and what each one looks like to a camera versus a hand.
The five tasks
| Task | What vision sees | What touch tells you | If it’s missed |
|---|---|---|---|
| Seating a connector (1D) | Looks in place, but the latch is hidden | The click: a sharp drop in force | A half-seated connector that fails later |
| Opening a poly bag (2D) | Clear film glares or is seen through | One layer or two, and whether it’s sliding | Two bags picked as one, or a torn bag |
| Pressing a foam insert (3D) | Light and crushed look the same | Pressure across the palm | Crushed product, a rejected kit |
| Holding a soft pouch (3D) | Slip is visible only once it falls | Fingertip slip, in time to re-grip | Dropped items, a stopped cell |
| Routing a cable (1D) | The path, but not tension or snags | Tension, snags and bend under the hand | A strained or pinched cable |
The labels 1D, 2D and 3D refer to the shape of the object: long and thin, flat and thin, or soft and bulky. We cover that in more detail in a separate field guide.
Seating a connector
A connector is small, and its latch sits inside the housing. From any practical camera angle, a fully seated connector and a half-seated one look almost the same.
The difference is easy to feel. As the latch engages, resistance builds and then drops sharply. That click is a clear signal in force, and it lasts only a moment.
Miss it and the failure is quiet. The connector holds, the system powers on, and some time later the connection drops. That is the worst kind of error: one you don’t find until it costs you.
Opening a poly bag
Clear film is hard for cameras. Depending on the light, it either glares or disappears. Two sheets stuck together look like one.
A fingertip resolves this quickly. Thickness, friction and the way layers slide against each other all tell you whether you are holding one layer or two, and whether the film is starting to move.
Miss it and you pick two bags as one, or pull too hard on a bag that has caught and tear it.
Pressing a foam insert
Foam compresses. Pressed lightly, it looks a certain way. Pressed too hard, it looks almost the same, until you release it and find the product underneath is damaged.
What tells them apart is pressure spread across the palm and fingers. That distribution says whether you are seating the insert or crushing it.
Miss it and you get crushed product and a rejected kit.
Holding a soft pouch
A soft pouch shifts in the hand as the arm moves. A camera only sees a slip once the pouch is visibly moving, which usually means once it is falling.
People catch slips much earlier. The human hand has about 17,000 mechanoreceptive units,1 and when a held object begins to slide, grip corrects in about 74 milliseconds.2 The signal is small shear at the fingertip, long before anything moves far enough to see.
Miss it and items drop, and a work cell stops while someone clears the floor.
Routing a cable
Vision can see the route a cable takes. It can’t see the tension along it, a snag behind a bracket, or how tightly it is bent under the hand.
Touch carries all three. Tension shows up as pull on the grip. A snag shows up as a sudden change in resistance. Bend shows up as the shape of contact along the fingers.
Miss it and you get a strained or pinched cable. Like the half-seated connector, that often works at first and fails later.
What the five have in common
Look across the table and one pattern holds. In every row, the important information is at the point of contact, and it is either hidden from the camera or visible only after the failure has already happened.
That is why we say the gap is at the point of contact, and video can’t see it.
Why force/torque isn’t enough
Many robot arms already carry a force/torque sensor at the wrist. It is useful, but it is not touch. It tells you how hard the arm is pressing in total. It doesn’t tell you where on the fingers contact is happening, or whether the object is slipping. Several of the rows above depend on exactly those two things.
What others have found
Other teams have published results that support adding touch for contact-heavy work. Liu et al. reported peg insertion success going from 45% to 90% with touch.3 Funk et al. reported robotic match-lighting going from 20% to 80% with touch.4 Figure moved its Helix policy from vision-only to fingertip touch.5
These are their results on their tasks. We cite them because they are the best public evidence on the question, not because they say anything about our own system.
How we are approaching it
We are building hands and a model designed to read these signals. Our next-generation AdiBot hands are designed to carry 48 tactile channels per finger, sampled at 100 Hz. Our VLSA model is designed to take those signals through a touch encoder, turn them into tokens alongside vision and language, and close the loop at 100 Hz on the robot.
The five tasks in the table are a working list for us. We are testing against them now. When we have results, we will publish them, including the ones that don’t go our way.