Hoofddorp, Netherlands, Wednesday, 20 January 2021 – According to the WHO, the number of people with some form of visual impairment is estimated to be as high as 2.2 billion people around the world (*1). With the aim of realizing a society where the visually impaired can live without any discomfort, five companies including OMRON Corporation, Alps Alpine Co. Ltd., Shimizu Corporation, IBM Japan, Ltd., and Mitsubishi Motors Corporation, which are specialized in different fields, have developed an "AI Suitcase" by combining their latest AI and robotics technologies. This article introduces an example of companies working together to improve accessibility (*2) for people with visual impairments and integrating technologies to meet the challenges of social implementation.
*1 Source: World report on vision: https://www.who.int/publications/i/item/world-report-on-vision
*2 Accessibility: ease of access to information and services
Enabling Free Movement and Natural Communication for the Visually Impaired
The number of visually impaired people is increasing every year due to accidents, loss of vision with aging and the development of eye diseases such as glaucoma. In Japan, there are an estimated 1.64 million visually impaired people, of which 188,000 are totally blind. (*3) These visually impaired people are inconvenienced in two main situations in their daily life. First is they cannot "move freely” since it takes more effort to check their route on a map, head to a destination, or walk down the street while avoiding people or objects. Second is the inability to read facial expressions which limits the communication features. Both are an essential part of our daily lives, such as going shopping, going to work or school, or traveling.
The person who attempted to use the power of technology to solve these problems is an IBM Fellow Dr. Chieko Asakawa, who lost her sight at age 14 and has a visual impairment herself. Dr. Asakwa is a researcher who has worked to improve accessibility of information for people with disabilities, including the development of the world's first website reading out software. Dr. Asakwa's policy is to "always implement technologies that she has studied to the society". Her subsequent research theme was the "AI Suitcase", an integrated solution for supporting independent mobility using AI, which aims to improve accessibility and quality of life for the visually impaired.
"People with visual impairments cannot walk around the city alone, nor enjoy themselves in peace. I want to solve these problems, but there are many hurdles to overcome to implement the technology in society. To get over these hurdles, it is necessary for companies in different industries to integrate their latest AI and robotics technologies" said Dr. Asakwa. The five companies that shared her thoughts established the "Consortium for Advanced Assistive Mobility Platform", a general incorporated association. The challenge was to develop an AI suitcase to support the mobility and communication of the visually impaired.
*3 Source: "Social cost of visual impairment in Japan," 2006-2008, Japanese Ophthalmologist Society study group
The Connection Between the "AI Suitcase" and OMRON's DNA
The "AI Suitcase" is an integrated solution consisting a set of a wearable device and a suitcase-style navigation robot that helps the visually impaired to move around the city independently. Information obtained from digital vision is supplemented by a combination of the latest AI and robotic technologies to support mobility and communication by recognizing the faces and expressions of people passing by, as well as their surroundings and the presence of obstacles. By making it into a suitcase, visually impaired people can carry it around with them in their daily lives without difficulty, and the system can provide them with tailored support at all times, depending on their circumstances.
OMRON's endorsement for this project was due to the fact that it embodies its founding DNA of solving social issues through its businesses and generating new innovations through technology. From OMRON, a team led by Kazuo Yamamoto of the Image Sensing Laboratory, Research and Development Center, Technology and Intellectual Property Division, worked on the development of the system by utilizing the knowledge OMRON has accumulated in the field of image recognition.
Natural Communication with the Ability of Sensing the Mood
OMRON is responsible for developing a function that uses facial image recognition technology to support smooth communication with people. This technology is a face-sensing technology that finds people you know and determines whether it is okay to talk to them or not.
Specifically, a camera on the user's shoulder spots a person approaching users about 5 meters away. The images from this camera are enhanced using AI to recognize the face of the approaching person in real time. When a person approaches, the system uses AI-based facial recognition to determine "Who that person is?". When a person comes closer to a certain distance, the system uses an expression estimation function to acquire information such as the person's facial expression, behaviour, and whether the person is looking at the user, and then determines "Whether the person is smiling or not?", "Is it okay to talk or not?". The goal is to establish a system that can give this information by notifying using voice and vibration (*4). By conveying more information to the visually impaired, the system will enable users to communicate naturally as they will be able to "read the mood".
Nevertheless, in reality, users move around indoors and outdoors. Depending on the location, the image from the camera is affected by light and brightness which makes it difficult to recognize faces and images. These technical problems need to be addressed before sensing can be made possible in a variety of everyday situations. The development team members are still working hard to clear these issues.
*4: Alps Electronics, a member of the consortium, utilizes its knowledge of tactile interfaces to guide users by transmitting information about who is presently in what state through auditory (voice) and tactile (vibration) signals.
Challenges for the Future that Continue Beyond the COVID-19 Pandemic
The new coronavirus has created another challenge in the development of this "read the mood" technology. It made difficult to recognize faces and facial expressions since everyone needs to wear a mask and ensure social distance. However, new technological developments have already begun to overcome these challenges.
Yamamoto, the team leader, describes his approach to technology development as follows, "As a researcher, I want to create new technologies that are one step ahead in order to solve people's concerns. Then I want to commercialize the technologies we have developed and implement them in society."
In the past, new communication tools, such as telephones and keyboards, have been created in response to the challenges and needs of people with disabilities, such as those who are blind or unable to use their hands freely. The technology developed in the "AI Suitcase" project will also help not only the visually impaired, but also those who need assistance to walk around on their own and those who need assistance with communication. It might lead to the development of robots that work collaboratively with people. We are moving forward to implementing the "AI Suitcase" in society, after conducting experiments.
Download PDF version here.