A Purdue University researcher and his team are developing technology to give robots the ability to learn language. A team led by Jeffrey Mark Siskind, associate professor in Purdue’s School of Electrical and Computer Engineering, has developed three algorithms that allow a wheeled robot to learn the meanings of words from example sentences that describe example paths taken by the robot, to use the words to generate a sentence to describe a path of movement, and to comprehend the sentence in order to produce a new path of movement.
“It’s our hope that this technology can be applied to a host of applications in the future, potentially including autonomous vehicles,” Siskind said.
The team took a small-wheeled robot outfitted with several cameras and ran numerous trials on an enclosed course containing several objects such as a chair, a traffic cone and a table. The sentences describing the path for the robot to take were provided by anonymous online sources. An operator then steered the robot to follow the paths described by the sentences. Using the algorithms, the robot was able to recognize words associated with objects within the course and words associated with directions of travel based on its sensory data.
“It was able to generate its own sentences to describe the paths it had taken. It was also able to generate its own sentences to describe a separate path of travel on the same course,” Siskind said. “The robot aggregated its sensory data over numerous experiences.”
By learning the meaning of various words, the robot took a step above conventional autonomous vehicles that work by using a vehicle’s electrical system to control driving based on a computerized map of existing roads. They also include cameras and various other sensors to detect potential hazards such as stoplights, pedestrians and the edges of the road. But current autonomous vehicles can’t recognize everyday landmarks off the road based on the sensory data. Nor can they associate words with the objects.
Siskind compares it to searching for a video on the internet. “You can search for a video of something you want to see online. But the search engine is not actually searching for the video. It’s searching for the captions under it, the words used to describe it,” he said. “What we’re doing with our research is actually recognizing what’s going on in the video.”
A 2015 grant from the National Robotics Initiative (NRI) paved the way for the research. The initiative seeks to accelerate the development and use of robots that work beside or cooperatively with people. Scott Bronikowski, then a doctoral candidate working with Siskind, developed the idea. He earned his doctorate in electrical and computer engineering from Purdue in 2016. Before attending Purdue, Bronikowski served multiple overseas tours, including in Iraq as an active duty U. S. Army officer. He retired from the Army in 2016.
“We do a lot with robots in the Army,” Bronikowski said. “But what we call robots really are just rugged remote control vehicles, and that means one soldier is controlling the robot and another soldier needs to watch over him or her. That makes the soldier vulnerable, and I thought if we had a robot that could be controlled by voice it would make things much easier and safer. I was thinking something like R2D2 from ‘Star Wars.’”
While the technology is still in its very early stages, Bronikowski believes that in addition to assisting the military and autonomous vehicles of the future, other potential applications for the technology include assisting people with limited functions perform everyday tasks and assisting emergency personnel in any disaster or dangerous situation.
“The Purdue technology is in a very basic stage at this point,” Bronikowski said. “But we could see a potential long-term application in autonomous vehicles.”
Purdue researchers are working to make the language-learning robot to scale, making it handle a wider range of situations and getting it to recognize a wider variety of words and phrases after hearing them for the first time.