Gestures and voice to move the crane

2008/11/01 Kortabitarte Egiguren, Irati - Elhuyar Zientzia Iturria: Elhuyar aldizkaria

• Teknologia

(Photo: L. Viatour)

The system is based on communication between humans and machines. InterAmi is a project that aims to move the crane with voice and gestures. To do this, the crane has two infrared cameras that collect the position and gestures of the operators. Operators will generally carry four retroreflective markers on arms, head and body parts. An infrared light focus will illuminate these markers and the two cameras that identify this type of light will receive the movements of the body, regulated internationally. For example, stretching the right arm means the crane must move in that direction.

In short, the cameras will collect images from which they will extract the information offered by infrared light markers. The points obtained in each image will serve to determine the spatial location of the user. In this way, the application can detect whether the user is making any gesture or not, and if a gesture is identified, the process would go ahead.

At the same time, operator instructions or voice commands will be collected by the application with the help of a laryngeal microphone, even in noisy environments. These microphones present greater immunity to ambient noise than conventional microphones.

Voice and an image

Infrared cameras collect the position and gestures of operators.

Robotiker-Tecnalia

Both voice and image signals complement each other, and the device does not perform any action if it does not receive the same order through both signals. In short, both signals are synchronized at all times. As a safety measure, the system compares the gesture and voice commands: if you notice that they give the same order, this signal will be sent to the crane and will fulfill the movement indicated by the crane.

The movement of the arms and the verbal order must occur simultaneously and at a certain time. Otherwise, the crane will not react. This can be said to reduce occupational risks. In fact, the system does not make any movement unless the user with the marker indicates it.

The system will focus on industrial facilities and collect information from staff in more than one area of the facility. Therefore, it must meet a number of requirements: on the one hand, the system must pay attention to the gestural commands of the employees, such as the position, the movement of the hands and the voice; on the other, the system must be able to work in multiple lighting conditions; and finally, the misunderstandings between the commands that the user facilitates and interprets will be avoided.

At the moment tests are being carried out in a virtual workshop.

Robotiker-Tecnalia

So far, Robotiker-Tecnalia researchers have developed a virtual installation and crane in a maritime freight port. The system knows all the areas where the crane moves, the location of all its elements and their possible movements. If the crane is detected to crash into an object, this application informs the user and the user automatically stops the activity.

While for the moment these applications have been incorporated into a virtual crane -- and only in a computer environment -- they hope to be able to apply them in future real cranes.