Recognising Human Gestures Without Cameras or Wearables

A new scientific publication by Dimitrios Tsiakmakis, Massimiliano Iosi, and Gastone Ciuti from Scuola Superiore Sant’Anna (SSSA), a partner of the MANiBOT project, has been published in IEEE Robotics and Automation Letters.

The article, “Non-Intrusive Contactless Gesture Recognition for Human-Robot Interaction,” explores how humans can communicate with robots through simple mid-air hand gestures, without relying on cameras or wearable devices.

The work is closely connected to MANiBOT’s broader vision of developing mobile service robots that can operate safely, naturally, and effectively in real-world environments where people are present, including retail, airport logistics, and other shared workspaces.

A more natural way to communicate with robots

As robots move beyond controlled laboratory settings and enter shared workplaces, they need to understand human intentions quickly and reliably. In environments such as baggage handling areas, retail spaces, hospitals, warehouses, and factories, operators may need to guide a robot, give a command, or interrupt an action without using a complex interface.

Gesture recognition offers an intuitive way to do this. A simple hand movement can communicate a command such as “move forward”, “stop”, “pick”, or “place”. However, many current systems depend on cameras, which can raise privacy concerns and may be affected by lighting, occlusions, or crowded environments. Other systems rely on wearable sensors, such as gloves or armbands, which can be impractical when many users interact with the same robot.

The article addresses this challenge by proposing a system that is contactless, privacy-preserving, and easy to integrate into shared environments.

How the system recognises gestures without cameras or wearables

The proposed system uses an array of capacitive-based proximity sensors placed beneath a surface. When a user performs a hand gesture above or near the surface, the sensors detect the movement without recording images. These sensor signals are then processed by a lightweight AI model, which classifies the gesture in real time.

The system recognises 16 gesture classes, including directional commands and task-oriented actions such as pick, place, hammer, and screw. This makes it relevant not only for basic robot navigation, but also for more complex collaborative tasks where a human operator may need to guide a robot during manipulation.

To reduce the amount of labelled data needed for training, the researchers used an active learning strategy. This allows the model to focus on the examples where it is most uncertain, making the training process more efficient and practical.

Strong performance for real-time interaction

The system achieved strong performance, reaching 97.2% accuracy across all 17 classes, including the 16 gestures and one non-gesture class. It also showed high precision and recall, outperforming the baseline methods tested in the study.

The system is also fast enough for real-time use, processing a gesture in around 16 milliseconds on a standard GPU. This is particularly important for human-robot interaction, where delays can make communication feel unnatural or reduce safety.

Another important finding is that the system continues to work when a non-conductive material, such as plastic or silicone, is placed over the sensors. This means that the technology could potentially be embedded into surfaces, workstations, robot skins, or protected interfaces without exposing the sensors directly.

The system also generalised well across different users, without requiring individual calibration.

Bringing gesture-based interaction closer to real-world use

The main value of this approach is that it makes human-robot communication more natural while avoiding some of the limitations of camera-based or wearable systems. Since no images are captured, the system supports privacy-preserving interaction. Since no wearable device is required, it is easier to use in settings where several people may need to interact with the same robot.

The system’s lightweight and modular nature also makes it promising for practical deployment. It could be integrated into tabletops, industrial workstations, collaborative robot interfaces, or even robotic skin. In this way, gesture recognition could become part of the physical environment, rather than an additional device that users need to wear or operate.

How this work connects to MANiBOT

MANiBOT aims to develop mobile service robots with advanced manipulation capabilities for demanding, real-world environments. The project focuses on robots that can handle diverse and partly unknown objects in a more human-like way, using advanced perception, tactile and proximity sensing, cognitive control, and bi-manual manipulation.

In MANiBOT’s use cases, including supermarket restocking and airport baggage handling, human operators may need to communicate with robots quickly and naturally. The contactless gesture recognition system developed by SSSA could support this need by providing an intuitive interface for commands related to navigation, manipulation, and task execution.

Gestures such as pick, place, increase, stop, or directional commands can map directly onto the types of robot behaviours required in MANiBOT’s scenarios. This makes the work highly relevant to the project’s broader goal of enabling safer, more trustworthy, and more efficient human-robot collaboration.

Learn more

The full publication, “Non-Intrusive Contactless Gesture Recognition for Human-Robot Interaction,” is available through IEEE Xplore:
https://ieeexplore.ieee.org/document/11491873