For decades, visions of the future have featured robots serving as domestic assistants. But despite many years of development in the robotics field, the dream of the domestic robot remains unfulfilled except for a few elementary tasks like floor vacuuming. Toyota Research Institute said it has developed a Soft Bubble Gripper toward that goal.

Robotic assistance can not only help clean homes and offices, but it can also assist older people or those with disabilities or age-related challenges so they can live with greater independence, dignity, and joy, said the Toyota Research Institute (TRI).

As part of its efforts toward a domestic robot, TRI has focused on improving robotic manipulation. Such grippers must be capable of stable grasps, precise placement, and safe interactions during inadvertent contact. They must also be low cost to be part of an affordable and commercially viable household robot. TRI said its engineers have designed a manipulator with all these capabilities called the Soft Bubble Gripper.

Soft Bubble Gripper builds on past advances

The Soft Bubble Gripper builds on past work by the TRI Manipulation Team. Previously, TRI’s robots gripped objects, sorted them, and even correctly placed items in a dishwasher using conventional, two-fingered grippers guided by external cameras. The grippers relied on these cameras to do their work. In other words, they had no sense of touch.

Now, TRI said it has found a way to improve robotic perception and manipulation with soft grippers that both passively hold objects better and actively sense how much force is applied. The soft grippers also accurately measure lateral force, which indicates when an object is about to slip out of a gripper’s grasp.

The gripper’s development team is led by Alex Alspach, who tapped into his soft robot development background to ideate the new gripper design, and Naveen Kuppuswamy, who formulated algorithms to use it. Together with TRI’s Tactile Team, they developed technology that uses the robust and compliant nature of air-filled, elastic bubbles for gripping with sensing from cameras on the inside. The cameras show what’s happening from a new perspective inside the grasp, including forces that are usually invisible.

Alspach and Kuppuswamy iterated their design, building on the work of other research organizations. Early designs used a robot arm with a single, large, round, soft-bubble on its end effector, which then graduated to two arms — a.k.a. “big fingers” — for dexterous, tactile-driven tasks like blindly sorting objects and threading a nut onto a giant bolt.

TRI said its researchers now use a single arm with two smaller fingers, each using a soft bubble which combines all the advantages of compliant gripping with real-time, real-world tactile sensing. The bubbles feel shapes and forces and recognize the object they are gripping, as well as the forces between the object and the fingers.

In addition, the team designed the Soft Bubble Gripper with inexpensive materials to eliminate a potential obstacle to domestic robot adoption.

Soft Bubble Gripper capabilities

The combination of force and shape data as visualized by these sensors allows robots equipped with Soft Bubble Grippers to perform a range of tasks that would be extremely difficult for rigid grippers to accomplish, according to the TRI researchers. Thus far, lab-validated capabilities of the Soft Bubble Gripper include the following innovations:

Robust and passively-compliant grasping. The Soft Bubble Gripper uses the inherently grippy texture and durable elastic properties of latex. The latex is inflated to a degree of softness that optimizes compliance to the shapes of held objects, maximizing the grasp stability. Given the physical properties — its air-filled bubbles and the gummy, high-friction texture of latex — the Soft Bubble Gripper is very reliable when it comes to grabbing and holding onto objects.

“Passive compliance” refers to grippers controlled by the laws of material physics rather than a motor and compliance to whatever shape is in the end effector. However, as Alspach noted, “compliance alone doesn’t let you do creative things with the object. This is where the gripper’s other capabilities come into play.”

Recognizing objects by touch. Inside the bubble is a low-cost, off-the-shelf, time-of-flight (ToF) depth sensor/infrared camera that uses vision to ‘feel’ what the gripper is holding. It enables the system to recognize objects by their shape and other physical properties and understand what to do with it within about a second.

This is analogous to human fingers searching for house keys buried deep in a purse or backpack. They can’t technically “see,” but they provide tactile information people use to build a mental model of what we are touching. This capability lets the robot perform realistic, useful tasks typically found in a home, because it understands how the object is oriented in its “hand” and what it must do to complete the task.

This is not much different from a toddler playing with a shape-sorter toy. When children presented with a pile of differently shaped blocks, they randomly grab one and feel its shape and all of its facets, and make their selections. The Soft Bubble Gripper uses a similar process to sort a sink full of objects. It measures and recognizes geometric features, then either precisely sets mugs in the dishwasher or drops plastic bottles into the recycling bin. It sorts the shapes, not visually, but through touch, according to TRI.

Shear force detection and interpretation. The Soft Bubble Gripper can also sense when some outside force is trying to take, twist, pull, or push an object, said the researchers. It uses the camera inside the bubbles to measure how a dense dot pattern inside the latex membrane is moving and distorting.

It then infers the magnitudes and directions of the forces causing this distortion. This lets it swiftly sense when the object it is lowering has landed on the counter, if it has accidentally bumped into something, or when an object it is handing over has been received. The end-of-arm tooling gives the robot tactile awareness of the outside world and its cohabitants. Depending on the controller, the robot could gently hand someone a full wine glass or set it on the table without spilling.

“To achieve a handover, the robot doesn’t need to know it is holding a wine glass, only that something is pulling on it … exerting an external force that tells it to let go,” explained Kuppuswamy.

In another exercise, the robot stacks transparent wine glasses. This would be incredibly difficult for a robot with conventional vision and hard grippers, because it is difficult to perceive transparent objects, and wine glasses are fragile. But it is relatively easy with the Soft Bubble Gripper, and the robot blindly stacks stemware with no knowledge of the object height or table position.

Shear force — read as directional changes in the dot pattern inside the latex membrane — indicates that one glass has successfully landed on top of another and tells the robot to let go.

“This kind of tactile sensing is absolutely necessary for the uncertainty that you see in people’s homes,” said Alspach. “We’re not in a controlled factory environment anymore. Our goal is to be helpful to people in their homes, and to do so we have to build a domestic robot with a sense of what it is touching.”

In addition, the robot is able to perceive and track changes in rotation of an object in its grasp. The robot can push the bottom of a mug against the kitchen counter to actually flip it into a better position for loading into the dishwasher.

“The ‘a-ha’ moment here is that instead of putting all our computing energy into how the robot approaches and picks up an object — which is a requirement for blind robots — we fast-forward past that to the actual task,” said Kuppuswamy. “The robot quickly grabs anything in the sink. Then we can focus on and adjust to what’s happening in its hand. This is quite different from a traditionally slow process needed for a factory robot.”

Operating blind. Most robots rely heavily on cameras to create a sense of vision. This means that light is needed to help create this sense of perception. The technical term for this is “visual occlusion.” Traditionally, robots also have difficulty working around clutter or in confined spaces, where there’s no clear line of sight. Transparent, shiny, or dark objects are also quite hard for robots to see.

However, since a robot equipped with the Soft Bubble Gripper relies on sensors inside the bubble and not an external camera to recognize and manipulate objects, it works equally well in lit or darkened rooms. The technology is suitable for situations in which it must reach into cluttered places (such as a sink full of miscellaneous objects and/or water) or has to manipulate in a way that would otherwise block its own view, said TRI.

“We’re not trying to replace external vision sensors — they’re still important,” Alspach noted. “We’re adding to these systems with even more useful sensory information resulting in increased capabilities and added precision.”

Training through self-annotation. Traditionally, a robotic system guided by computer vision is trained to recognize objects by repeatedly showing it images to establish defined categories — a time-consuming process known as supervised training.

For example, somebody has to show the robot many times over various photos of coffee mugs and tell it, “These are all coffee mugs.”

The team TRI team claimed that it had a breakthrough when it set up the robot to self-annotate by filling a sink with one type of object and letting the robot repeatedly grasp and drop these objects, reducing the amount of time it took to “learn” a new object. This recognition process is repeated for every new object the team wants the Soft Bubble Gripper to learn. As a result, tactile sensors must be designed to withstand the many cycles needed for this type of training

“By cutting the human out of the loop completely, the grasp data was a better match to the variability the robot would experience in real-world situations,” added Alspach.

Use of low-cost materials. Toyota said its team used simple fabrication techniques and affordable materials so that the Soft Bubble Gripper would be inexpensive to build, operate, maintain, and repair.

“An insight we gained is finding a way to make an inexpensive camera deliver usable depth data at an extremely close range,” Alspach said. “It’s how we angled the camera that lets us maximize the quality of the depth measurements — so that the camera can view and identify the ‘tactile’ imprint of the gripped object on the soft latex bubble.”

Goals for future TRI exploration

A long-term goal of the TRI robotics research team is to break the traditional robot controller paradigm and accommodate for an ever-changing environment.

Improving reflexes. The team said it wants to replicate real-time course-correcting behavior in robots. It is striving for distributed robotic sensing where information is processed and reacted to “locally” instead of having to refer to a centralized ‘brain’ for every decision. This enables robots to execute their actions much quicker with less forethought.

“It’s more like what humans or animals do,” Alspach explained. “In a fraction of a second, we react and catch the thing that fell off the table or slipped out of our hand almost without thinking.”

Expanding softness. Currently, only the robot gripper is soft. But what if more of its surface could be made of such materials?
That could bring great advantages for human/robot interaction including working in the home with children, pets or people who may already have physical challenges.

“We aim to build robots that can interact with the world around them comfortably,” said Alspach. “Softness makes the application of robots more reasonable for homes. Taking it further, we can begin to incorporate lightweight materials, air, smaller electronics and an optimized design to help us reduce weight. It’s the combination of soft and lightweight, along with the ability to sense and react, that contributes to safety.”

“It’s actually a really tricky problem,” added Kuppuswamy. “In one sense, it’s extremely hard to build because you are working with frontier materials…. It’s not just classical methods and materials like machining or welding.”

Further complicating this is the fact that anything inflatable introduces the factor of continuously dynamic shape-shifting, which destabilizes the usual equations. “Mathematically, soft is hard,” Kuppuswamy said.

Publication, recognition on the way to more useful robots

More information about the TRI Soft Bubble Gripper research can be found in the published paper, “Soft Bubble grippers for robust and perceptive manipulation.” The team’s research was also recognized at ICRA 2020, the annual IEEE International Conference on Robotics and Automation, with a 2019 Robotics and Automation Letters Best Paper award for another publication about the soft gripper project.

TRI welcomes other scientists researching robotic manipulation to build on their work by bringing their own insights and discoveries. To inspire this collaboration and further the research on soft bubble grippers, they hosted VisuoTactile 2020, a workshop of thought-leaders in visuo-tactile technology at the Robotics Science and Systems (RSS) conference.

The company said the Soft Bubble Gripper builds on its manipulation research toward making human-assist robots reliable and robust. Even without the sensing capability, the stretchy material or low stiffness makes for a superior gripper in comparison with with standard soft grippers, said the researchers. It can conform to a wide variety of shapes and get a stable grasp. The gripper combines strength and smarts to handle objects that require a delicate touch and a steady hand, said the team.

Through sensing geometry, camera images, and shear forces, robots can now perceive the objects they are holding by estimating the pose of objects and sensing forces on the surface. Because the Soft Bubble Gripper uses relatively inexpensive cameras internally, it moves manipulation for applications such as domestic robots a step closer to reality, said TRI.